Oct 30 19:15:38.072094 ip-10-0-137-84 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 19:15:38.474578 ip-10-0-137-84 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:15:38.474578 ip-10-0-137-84 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 19:15:38.474578 ip-10-0-137-84 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:15:38.474578 ip-10-0-137-84 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:15:38.479937 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.479858    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:15:38.479937 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.479938    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:15:38.480004 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.479944    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:15:38.480004 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.479948    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:15:38.480004 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.479952    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:15:38.480004 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.479955    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:15:38.480004 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.479958    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:15:38.480004 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.479962    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:15:38.480004 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.479966    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:15:38.480004 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.479969    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:15:38.480004 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.479972    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:15:38.480004 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.479975    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:15:38.480004 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.479977    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:15:38.480004 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.479980    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:15:38.480004 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.479982    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:15:38.480004 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.479992    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:15:38.480004 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.479995    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:15:38.480004 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.479998    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:15:38.480004 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480000    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:15:38.480004 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480003    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:15:38.480004 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480006    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:15:38.480444 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480009    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:15:38.480444 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480011    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:15:38.480444 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480014    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:15:38.480444 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480017    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:15:38.480444 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480020    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:15:38.480444 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480023    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:15:38.480444 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480025    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:15:38.480444 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480028    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:15:38.480444 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480030    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:15:38.480444 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480033    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:15:38.480444 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480035    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:15:38.480444 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480038    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:15:38.480444 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480040    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:15:38.480444 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480043    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:15:38.480444 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480046    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:15:38.480444 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480049    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:15:38.480444 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480051    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:15:38.480444 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480054    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:15:38.480444 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480056    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:15:38.480444 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480059    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:15:38.480924 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480062    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:15:38.480924 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480065    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:15:38.480924 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480067    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:15:38.480924 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480070    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:15:38.480924 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480072    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:15:38.480924 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480075    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:15:38.480924 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480078    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:15:38.480924 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480082    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:15:38.480924 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480085    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:15:38.480924 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480088    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:15:38.480924 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480090    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:15:38.480924 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480093    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:15:38.480924 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480096    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:15:38.480924 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480098    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:15:38.480924 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480101    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:15:38.480924 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480104    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:15:38.480924 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480106    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:15:38.480924 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480109    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:15:38.480924 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480111    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:15:38.480924 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480114    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:15:38.481384 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480117    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:15:38.481384 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480119    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:15:38.481384 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480122    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:15:38.481384 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480124    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:15:38.481384 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480127    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:15:38.481384 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480130    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:15:38.481384 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480132    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:15:38.481384 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480134    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:15:38.481384 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480137    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:15:38.481384 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480139    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:15:38.481384 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480142    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:15:38.481384 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480145    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:15:38.481384 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480147    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:15:38.481384 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480149    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:15:38.481384 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480152    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:15:38.481384 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480154    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:15:38.481384 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480157    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:15:38.481384 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480159    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:15:38.481384 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480162    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:15:38.481384 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480164    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:15:38.481898 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480168    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:15:38.481898 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480171    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:15:38.481898 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480173    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:15:38.481898 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480176    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:15:38.481898 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480178    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:15:38.481898 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480181    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:15:38.481898 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480184    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:15:38.481898 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480186    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:15:38.481898 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480189    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:15:38.481898 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480191    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:15:38.481898 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480194    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:15:38.481898 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480196    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:15:38.481898 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480199    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:15:38.481898 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480201    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:15:38.481898 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480204    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:15:38.481898 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480206    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:15:38.481898 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480209    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:15:38.481898 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480211    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:15:38.481898 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480215    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:15:38.482379 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480217    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:15:38.482379 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480220    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:15:38.482379 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480223    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:15:38.482379 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480225    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:15:38.482379 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480227    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:15:38.482379 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.480230    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:15:38.482379 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481018    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:15:38.482379 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481025    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:15:38.482379 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481028    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:15:38.482379 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481031    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:15:38.482379 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481034    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:15:38.482379 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481037    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:15:38.482379 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481039    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:15:38.482379 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481042    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:15:38.482379 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481045    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:15:38.482379 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481048    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:15:38.482379 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481051    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:15:38.482379 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481054    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:15:38.482379 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481057    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:15:38.482379 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481060    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:15:38.482869 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481062    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:15:38.482869 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481065    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:15:38.482869 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481068    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:15:38.482869 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481070    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:15:38.482869 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481073    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:15:38.482869 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481075    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:15:38.482869 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481078    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:15:38.482869 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481081    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:15:38.482869 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481083    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:15:38.482869 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481086    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:15:38.482869 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481088    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:15:38.482869 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481091    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:15:38.482869 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481093    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:15:38.482869 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481096    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:15:38.482869 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481098    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:15:38.482869 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481101    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:15:38.482869 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481103    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:15:38.482869 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481106    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:15:38.482869 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481108    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:15:38.482869 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481111    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:15:38.483337 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481114    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:15:38.483337 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481116    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:15:38.483337 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481119    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:15:38.483337 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481121    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:15:38.483337 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481123    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:15:38.483337 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481127    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:15:38.483337 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481129    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:15:38.483337 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481134    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:15:38.483337 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481138    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:15:38.483337 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481141    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:15:38.483337 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481145    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:15:38.483337 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481147    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:15:38.483337 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481150    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:15:38.483337 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481153    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:15:38.483337 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481155    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:15:38.483337 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481158    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:15:38.483337 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481160    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:15:38.483337 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481163    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:15:38.483337 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481165    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:15:38.483806 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481168    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:15:38.483806 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481170    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:15:38.483806 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481172    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:15:38.483806 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481175    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:15:38.483806 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481177    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:15:38.483806 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481180    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:15:38.483806 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481183    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:15:38.483806 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481185    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:15:38.483806 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481187    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:15:38.483806 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481191    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:15:38.483806 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481195    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:15:38.483806 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481198    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:15:38.483806 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481201    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:15:38.483806 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481204    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:15:38.483806 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481206    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:15:38.483806 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481208    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:15:38.483806 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481211    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:15:38.483806 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481214    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:15:38.483806 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481217    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:15:38.484286 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481219    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:15:38.484286 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481222    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:15:38.484286 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481225    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:15:38.484286 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481228    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:15:38.484286 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481231    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:15:38.484286 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481234    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:15:38.484286 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481237    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:15:38.484286 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481239    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:15:38.484286 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481242    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:15:38.484286 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481244    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:15:38.484286 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481247    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:15:38.484286 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481249    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:15:38.484286 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481252    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:15:38.484286 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481254    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:15:38.484286 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481256    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:15:38.484286 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481259    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:15:38.484286 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481261    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:15:38.484286 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481264    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:15:38.484286 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481266    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:15:38.484286 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481269    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:15:38.484775 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481271    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:15:38.484775 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481275    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:15:38.484775 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481278    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:15:38.484775 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481281    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:15:38.484775 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481284    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:15:38.484775 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481287    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:15:38.484775 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481289    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:15:38.484775 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481292    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:15:38.484775 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481295    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:15:38.484775 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481297    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:15:38.484775 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481299    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:15:38.484775 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481302    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:15:38.484775 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481304    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:15:38.484775 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481307    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:15:38.484775 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481810    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:15:38.484775 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481816    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:15:38.484775 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481819    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:15:38.484775 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481822    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:15:38.484775 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481825    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:15:38.485218 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481828    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:15:38.485218 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481831    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:15:38.485218 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481833    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:15:38.485218 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481836    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:15:38.485218 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481839    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:15:38.485218 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481842    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:15:38.485218 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481845    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:15:38.485218 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481848    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:15:38.485218 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481850    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:15:38.485218 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481853    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:15:38.485218 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481856    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:15:38.485218 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481858    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:15:38.485218 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481861    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:15:38.485218 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481864    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:15:38.485218 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481867    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:15:38.485218 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481869    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:15:38.485218 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481871    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:15:38.485218 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481876    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:15:38.485218 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481879    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:15:38.485218 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481882    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:15:38.485730 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481884    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:15:38.485730 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481887    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:15:38.485730 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481889    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:15:38.485730 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481892    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:15:38.485730 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481894    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:15:38.485730 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481897    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:15:38.485730 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481901    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:15:38.485730 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481905    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:15:38.485730 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481909    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:15:38.485730 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481912    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:15:38.485730 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481916    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:15:38.485730 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481919    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:15:38.485730 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481922    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:15:38.485730 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481925    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:15:38.485730 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481927    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:15:38.485730 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481930    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:15:38.485730 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481932    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:15:38.485730 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481935    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:15:38.485730 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481937    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:15:38.486207 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481940    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:15:38.486207 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481942    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:15:38.486207 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481946    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:15:38.486207 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481948    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:15:38.486207 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481951    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:15:38.486207 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481953    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:15:38.486207 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481956    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:15:38.486207 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481958    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:15:38.486207 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481961    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:15:38.486207 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481964    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:15:38.486207 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481967    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:15:38.486207 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481970    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:15:38.486207 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481973    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:15:38.486207 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481975    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:15:38.486207 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481978    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:15:38.486207 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481980    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:15:38.486207 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481983    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:15:38.486207 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481985    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:15:38.486207 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481988    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:15:38.486688 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481990    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:15:38.486688 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481993    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:15:38.486688 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481995    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:15:38.486688 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.481998    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:15:38.486688 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.482001    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:15:38.486688 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.482004    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:15:38.486688 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.482006    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:15:38.486688 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.482008    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:15:38.486688 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.482011    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:15:38.486688 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.482013    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:15:38.486688 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.482016    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:15:38.486688 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.482018    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:15:38.486688 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.482020    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:15:38.486688 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.482023    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:15:38.486688 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.482025    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:15:38.486688 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.482028    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:15:38.486688 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.482031    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:15:38.486688 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.482033    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:15:38.486688 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.482036    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:15:38.486688 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.482038    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:15:38.487176 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.482041    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:15:38.487176 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.482043    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:15:38.487176 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.482045    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:15:38.487176 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.482049    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:15:38.487176 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.482054    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:15:38.487176 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.482058    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:15:38.487176 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.482060    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:15:38.487176 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.482063    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:15:38.487176 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.482065    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:15:38.487176 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.482068    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:15:38.487176 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.482070    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:15:38.487176 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.482073    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:15:38.487176 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.482075    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:15:38.487176 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.482078    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:15:38.487176 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.482080    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:15:38.487176 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.482083    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:15:38.487176 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.482086    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:15:38.487176 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.482088    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:15:38.487176 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.482091    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:15:38.487633 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.482093    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:15:38.487633 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.482096    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:15:38.487633 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.482098    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:15:38.487633 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.482101    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:15:38.487633 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483157    2579 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 19:15:38.487633 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483167    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 19:15:38.487633 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483173    2579 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 19:15:38.487633 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483179    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 19:15:38.487633 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483184    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 19:15:38.487633 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483187    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 19:15:38.487633 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483191    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 19:15:38.487633 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483195    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 19:15:38.487633 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483198    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 19:15:38.487633 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483201    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 19:15:38.487633 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483204    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 19:15:38.487633 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483208    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 19:15:38.487633 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483212    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 19:15:38.487633 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483215    2579 flags.go:64] FLAG: --cgroup-root=""
Oct 30 19:15:38.487633 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483218    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 19:15:38.487633 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483221    2579 flags.go:64] FLAG: --client-ca-file=""
Oct 30 19:15:38.487633 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483223    2579 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 19:15:38.487633 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483226    2579 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 19:15:38.487633 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483231    2579 flags.go:64] FLAG: --cluster-domain=""
Oct 30 19:15:38.487633 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483234    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 19:15:38.488215 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483237    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 19:15:38.488215 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483240    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 19:15:38.488215 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483243    2579 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 19:15:38.488215 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483247    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 19:15:38.488215 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483250    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 19:15:38.488215 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483253    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 19:15:38.488215 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483257    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 19:15:38.488215 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483260    2579 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 19:15:38.488215 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483264    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 19:15:38.488215 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483267    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 19:15:38.488215 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483270    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 19:15:38.488215 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483273    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 19:15:38.488215 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483277    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 19:15:38.488215 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483280    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 19:15:38.488215 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483283    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 19:15:38.488215 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483285    2579 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 19:15:38.488215 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483288    2579 flags.go:64] FLAG: --enable-server="true"
Oct 30 19:15:38.488215 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483291    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 19:15:38.488215 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483295    2579 flags.go:64] FLAG: --event-burst="100"
Oct 30 19:15:38.488215 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483298    2579 flags.go:64] FLAG: --event-qps="50"
Oct 30 19:15:38.488215 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483301    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 19:15:38.488215 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483305    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 19:15:38.488215 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483308    2579 flags.go:64] FLAG: --eviction-hard=""
Oct 30 19:15:38.488215 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483312    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 19:15:38.488862 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483315    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 19:15:38.488862 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483318    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 19:15:38.488862 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483321    2579 flags.go:64] FLAG: --eviction-soft=""
Oct 30 19:15:38.488862 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483324    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 19:15:38.488862 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483327    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 19:15:38.488862 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483330    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 19:15:38.488862 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483333    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 19:15:38.488862 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483336    2579 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 19:15:38.488862 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483338    2579 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 19:15:38.488862 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483342    2579 flags.go:64] FLAG: --feature-gates=""
Oct 30 19:15:38.488862 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483346    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 19:15:38.488862 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483349    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 19:15:38.488862 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483352    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 19:15:38.488862 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483356    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 19:15:38.488862 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483358    2579 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 19:15:38.488862 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483362    2579 flags.go:64] FLAG: --help="false"
Oct 30 19:15:38.488862 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483365    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-137-84.us-east-2.compute.internal"
Oct 30 19:15:38.488862 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483368    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 19:15:38.488862 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483371    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 19:15:38.488862 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483374    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 19:15:38.488862 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483377    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 19:15:38.488862 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483381    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 19:15:38.488862 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483383    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 19:15:38.489402 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483386    2579 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 19:15:38.489402 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483389    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 19:15:38.489402 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483391    2579 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 19:15:38.489402 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483394    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 19:15:38.489402 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483397    2579 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 19:15:38.489402 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483400    2579 flags.go:64] FLAG: --kube-reserved=""
Oct 30 19:15:38.489402 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483403    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 19:15:38.489402 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483406    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 19:15:38.489402 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483409    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 19:15:38.489402 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483412    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 19:15:38.489402 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483415    2579 flags.go:64] FLAG: --lock-file=""
Oct 30 19:15:38.489402 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483418    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 19:15:38.489402 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483421    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 19:15:38.489402 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483424    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 19:15:38.489402 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483429    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 19:15:38.489402 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483432    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 19:15:38.489402 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483435    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 19:15:38.489402 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483438    2579 flags.go:64] FLAG: --logging-format="text"
Oct 30 19:15:38.489402 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483441    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 19:15:38.489402 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483444    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 19:15:38.489402 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483447    2579 flags.go:64] FLAG: --manifest-url=""
Oct 30 19:15:38.489402 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483450    2579 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 19:15:38.489402 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483454    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 19:15:38.489402 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483457    2579 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 19:15:38.489402 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483461    2579 flags.go:64] FLAG: --max-pods="110"
Oct 30 19:15:38.490038 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483465    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 19:15:38.490038 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483468    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 19:15:38.490038 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483471    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 19:15:38.490038 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483474    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 19:15:38.490038 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483477    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 19:15:38.490038 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483479    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 19:15:38.490038 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483482    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 19:15:38.490038 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483489    2579 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 19:15:38.490038 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483492    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 19:15:38.490038 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483495    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 19:15:38.490038 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483498    2579 flags.go:64] FLAG: --pod-cidr=""
Oct 30 19:15:38.490038 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483501    2579 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 19:15:38.490038 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483504    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 19:15:38.490038 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483507    2579 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 19:15:38.490038 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483510    2579 flags.go:64] FLAG: --port="10250"
Oct 30 19:15:38.490038 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483513    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 19:15:38.490038 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483516    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-02388e4b8a9972580"
Oct 30 19:15:38.490038 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483519    2579 flags.go:64] FLAG: --qos-reserved=""
Oct 30 19:15:38.490038 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483529    2579 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 19:15:38.490038 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483534    2579 flags.go:64] FLAG: --register-node="true"
Oct 30 19:15:38.490038 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483537    2579 flags.go:64] FLAG: --register-with-taints=""
Oct 30 19:15:38.490038 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483541    2579 flags.go:64] FLAG: --registry-burst="10"
Oct 30 19:15:38.490038 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483544    2579 flags.go:64] FLAG: --registry-qps="5"
Oct 30 19:15:38.490038 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483547    2579 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 19:15:38.490038 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483550    2579 flags.go:64] FLAG: --reserved-memory=""
Oct 30 19:15:38.490038 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483554    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 19:15:38.490752 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483557    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 19:15:38.490752 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483560    2579 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 19:15:38.490752 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483563    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 19:15:38.490752 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483565    2579 flags.go:64] FLAG: --runonce="false"
Oct 30 19:15:38.490752 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483568    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 19:15:38.490752 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483571    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 19:15:38.490752 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483574    2579 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 19:15:38.490752 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483578    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 19:15:38.490752 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483581    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 19:15:38.490752 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483584    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 19:15:38.490752 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483587    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 19:15:38.490752 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483590    2579 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 19:15:38.490752 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483593    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 19:15:38.490752 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483595    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 19:15:38.490752 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483599    2579 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 19:15:38.490752 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483601    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 19:15:38.490752 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483604    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 19:15:38.490752 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483609    2579 flags.go:64] FLAG: --system-cgroups=""
Oct 30 19:15:38.490752 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483612    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 19:15:38.490752 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483617    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 19:15:38.490752 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483620    2579 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 19:15:38.490752 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483623    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 19:15:38.490752 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483628    2579 flags.go:64] FLAG: --tls-min-version=""
Oct 30 19:15:38.490752 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483630    2579 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 19:15:38.490752 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483633    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 19:15:38.491661 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483636    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 19:15:38.491661 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483639    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 19:15:38.491661 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483642    2579 flags.go:64] FLAG: --v="2"
Oct 30 19:15:38.491661 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483647    2579 flags.go:64] FLAG: --version="false"
Oct 30 19:15:38.491661 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483651    2579 flags.go:64] FLAG: --vmodule=""
Oct 30 19:15:38.491661 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483655    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 19:15:38.491661 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.483658    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 19:15:38.491661 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483830    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:15:38.491661 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483834    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:15:38.491661 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483837    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:15:38.491661 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483840    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:15:38.491661 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483843    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:15:38.491661 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483846    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:15:38.491661 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483848    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:15:38.491661 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483851    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:15:38.491661 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483854    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:15:38.491661 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483857    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:15:38.491661 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483859    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:15:38.491661 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483861    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:15:38.491661 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483864    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:15:38.491661 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483866    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:15:38.492192 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483869    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:15:38.492192 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483871    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:15:38.492192 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483874    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:15:38.492192 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483879    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:15:38.492192 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483882    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:15:38.492192 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483889    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:15:38.492192 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483891    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:15:38.492192 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483894    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:15:38.492192 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483897    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:15:38.492192 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483900    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:15:38.492192 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483903    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:15:38.492192 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483905    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:15:38.492192 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483908    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:15:38.492192 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483911    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:15:38.492192 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483913    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:15:38.492192 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483916    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:15:38.492192 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483918    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:15:38.492192 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483921    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:15:38.492192 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483924    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:15:38.492657 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483926    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:15:38.492657 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483929    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:15:38.492657 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483931    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:15:38.492657 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483933    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:15:38.492657 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483936    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:15:38.492657 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483939    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:15:38.492657 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483941    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:15:38.492657 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483944    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:15:38.492657 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483947    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:15:38.492657 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483949    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:15:38.492657 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483952    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:15:38.492657 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483955    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:15:38.492657 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483957    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:15:38.492657 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483960    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:15:38.492657 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483962    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:15:38.492657 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483965    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:15:38.492657 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483968    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:15:38.492657 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483972    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:15:38.492657 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483977    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:15:38.492657 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483979    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:15:38.493146 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483982    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:15:38.493146 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483985    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:15:38.493146 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483987    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:15:38.493146 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483990    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:15:38.493146 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483992    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:15:38.493146 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483995    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:15:38.493146 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.483998    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:15:38.493146 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484001    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:15:38.493146 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484004    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:15:38.493146 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484007    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:15:38.493146 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484009    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:15:38.493146 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484012    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:15:38.493146 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484014    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:15:38.493146 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484017    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:15:38.493146 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484019    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:15:38.493146 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484022    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:15:38.493146 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484024    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:15:38.493146 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484027    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:15:38.493146 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484029    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:15:38.493632 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484032    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:15:38.493632 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484034    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:15:38.493632 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484037    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:15:38.493632 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484040    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:15:38.493632 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484043    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:15:38.493632 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484047    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:15:38.493632 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484050    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:15:38.493632 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484052    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:15:38.493632 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484055    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:15:38.493632 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484058    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:15:38.493632 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484061    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:15:38.493632 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484064    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:15:38.493632 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484067    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:15:38.493632 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484069    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:15:38.493632 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484072    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:15:38.493632 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484075    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:15:38.493632 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484077    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:15:38.493632 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484080    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:15:38.493632 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484082    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:15:38.494112 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484085    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:15:38.494112 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484087    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:15:38.494112 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484090    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:15:38.494112 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484092    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:15:38.494112 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484095    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:15:38.494112 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484097    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:15:38.494112 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484100    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:15:38.494112 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484103    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:15:38.494112 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484105    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:15:38.494112 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484108    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:15:38.494112 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484111    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:15:38.494112 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484113    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:15:38.494112 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484116    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:15:38.494112 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484118    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:15:38.494112 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484121    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:15:38.494112 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.484147    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:15:38.494112 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484184    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:15:38.494513 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484188    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:15:38.494513 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484191    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:15:38.494513 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484194    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:15:38.494513 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484196    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:15:38.494513 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484199    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:15:38.494513 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484201    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:15:38.494513 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484205    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:15:38.494513 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484209    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:15:38.494513 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484211    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:15:38.494513 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484214    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:15:38.494513 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484217    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:15:38.494513 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484219    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:15:38.494513 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484223    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:15:38.494513 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484227    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:15:38.494513 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484230    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:15:38.494513 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484233    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:15:38.494513 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484235    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:15:38.494513 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484238    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:15:38.494513 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484240    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:15:38.494989 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484243    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:15:38.494989 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484245    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:15:38.494989 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484248    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:15:38.494989 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484250    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:15:38.494989 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484253    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:15:38.494989 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484256    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:15:38.494989 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484258    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:15:38.494989 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484261    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:15:38.494989 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484263    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:15:38.494989 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484265    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:15:38.494989 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484268    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:15:38.494989 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484271    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:15:38.494989 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484273    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:15:38.494989 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484282    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:15:38.494989 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484284    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:15:38.494989 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484287    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:15:38.494989 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484290    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:15:38.494989 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484293    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:15:38.494989 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484296    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:15:38.494989 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484300    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:15:38.495460 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484303    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:15:38.495460 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484306    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:15:38.495460 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484309    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:15:38.495460 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484312    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:15:38.495460 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484315    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:15:38.495460 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484317    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:15:38.495460 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484320    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:15:38.495460 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484322    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:15:38.495460 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484325    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:15:38.495460 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484327    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:15:38.495460 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484330    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:15:38.495460 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484332    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:15:38.495460 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484335    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:15:38.495460 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484337    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:15:38.495460 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484339    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:15:38.495460 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484342    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:15:38.495460 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484344    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:15:38.495460 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484347    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:15:38.495460 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484349    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:15:38.495460 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484352    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:15:38.495950 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484354    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:15:38.495950 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484357    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:15:38.495950 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484359    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:15:38.495950 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484362    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:15:38.495950 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484364    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:15:38.495950 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484368    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:15:38.495950 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484371    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:15:38.495950 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484374    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:15:38.495950 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484377    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:15:38.495950 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484379    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:15:38.495950 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484381    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:15:38.495950 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484386    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:15:38.495950 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484389    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:15:38.495950 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484392    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:15:38.495950 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484394    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:15:38.495950 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484397    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:15:38.495950 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484399    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:15:38.495950 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484402    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:15:38.495950 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484405    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:15:38.496390 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484407    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:15:38.496390 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484410    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:15:38.496390 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484412    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:15:38.496390 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484415    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:15:38.496390 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484417    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:15:38.496390 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484420    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:15:38.496390 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484422    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:15:38.496390 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484425    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:15:38.496390 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484428    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:15:38.496390 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484430    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:15:38.496390 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484433    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:15:38.496390 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484435    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:15:38.496390 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484438    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:15:38.496390 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484440    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:15:38.496390 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484444    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:15:38.496390 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484448    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:15:38.496390 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484451    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:15:38.496390 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484454    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:15:38.496390 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484456    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:15:38.496961 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484458    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:15:38.496961 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484461    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:15:38.496961 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484464    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:15:38.496961 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484467    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:15:38.496961 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484470    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:15:38.496961 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484474    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:15:38.496961 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484477    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:15:38.496961 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.484480    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:15:38.496961 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.492678    2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 19:15:38.496961 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.492696    2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 19:15:38.496961 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.493255    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 19:15:38.496961 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.493262    2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 19:15:38.496961 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493324    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:15:38.496961 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493329    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:15:38.496961 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493332    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:15:38.496961 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493335    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:15:38.496961 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493338    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:15:38.496961 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493341    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:15:38.496961 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493344    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:15:38.497419 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493346    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:15:38.497419 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493349    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:15:38.497419 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493352    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:15:38.497419 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493354    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:15:38.497419 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493357    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:15:38.497419 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493360    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:15:38.497419 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493362    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:15:38.497419 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493365    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:15:38.497419 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493368    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:15:38.497419 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493370    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:15:38.497419 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493373    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:15:38.497419 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493375    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:15:38.497419 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493378    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:15:38.497419 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493380    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:15:38.497419 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493383    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:15:38.497419 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493385    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:15:38.497419 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493388    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:15:38.497419 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493390    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:15:38.497419 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493393    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:15:38.497885 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493403    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:15:38.497885 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493406    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:15:38.497885 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493408    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:15:38.497885 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493411    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:15:38.497885 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493413    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:15:38.497885 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493416    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:15:38.497885 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493419    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:15:38.497885 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493421    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:15:38.497885 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493423    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:15:38.497885 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493426    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:15:38.497885 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493429    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:15:38.497885 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493431    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:15:38.497885 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493434    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:15:38.497885 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493437    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:15:38.497885 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493440    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:15:38.497885 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493442    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:15:38.497885 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493444    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:15:38.497885 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493447    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:15:38.497885 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493450    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:15:38.497885 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493452    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:15:38.498336 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493456    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:15:38.498336 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493459    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:15:38.498336 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493462    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:15:38.498336 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493464    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:15:38.498336 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493467    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:15:38.498336 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493469    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:15:38.498336 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493472    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:15:38.498336 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493474    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:15:38.498336 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493477    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:15:38.498336 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493481    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:15:38.498336 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493485    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:15:38.498336 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493489    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:15:38.498336 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493492    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:15:38.498336 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493494    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:15:38.498336 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493497    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:15:38.498336 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493499    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:15:38.498336 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493502    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:15:38.498336 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493505    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:15:38.498336 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493507    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:15:38.498797 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493510    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:15:38.498797 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493512    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:15:38.498797 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493515    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:15:38.498797 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493517    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:15:38.498797 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493521    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:15:38.498797 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493523    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:15:38.498797 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493526    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:15:38.498797 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493528    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:15:38.498797 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493531    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:15:38.498797 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493533    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:15:38.498797 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493536    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:15:38.498797 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493539    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:15:38.498797 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493543    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:15:38.498797 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493546    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:15:38.498797 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493550    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:15:38.498797 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493553    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:15:38.498797 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493555    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:15:38.498797 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493558    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:15:38.498797 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493561    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:15:38.498797 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493563    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:15:38.499310 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493566    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:15:38.499310 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493568    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:15:38.499310 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493571    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:15:38.499310 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493573    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:15:38.499310 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493576    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:15:38.499310 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493579    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:15:38.499310 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493582    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:15:38.499310 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493584    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:15:38.499310 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493587    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:15:38.499310 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493589    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:15:38.499310 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493592    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:15:38.499310 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493595    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:15:38.499310 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493597    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:15:38.499310 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493600    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:15:38.499310 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493602    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:15:38.499310 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493605    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:15:38.499310 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493607    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:15:38.499310 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493610    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:15:38.499310 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493612    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:15:38.499310 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493615    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:15:38.499804 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493617    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:15:38.499804 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.493646    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:15:38.499804 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493812    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:15:38.499804 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493817    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:15:38.499804 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493820    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:15:38.499804 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493823    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:15:38.499804 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493825    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:15:38.499804 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493829    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:15:38.499804 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493832    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:15:38.499804 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493834    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:15:38.499804 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493837    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:15:38.499804 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493839    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:15:38.499804 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493842    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:15:38.499804 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493844    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:15:38.499804 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493847    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:15:38.499804 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493849    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:15:38.500194 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493852    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:15:38.500194 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493855    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:15:38.500194 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493857    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:15:38.500194 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493860    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:15:38.500194 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493863    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:15:38.500194 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493865    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:15:38.500194 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493868    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:15:38.500194 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493870    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:15:38.500194 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493873    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:15:38.500194 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493876    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:15:38.500194 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493879    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:15:38.500194 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493881    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:15:38.500194 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493884    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:15:38.500194 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493886    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:15:38.500194 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493889    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:15:38.500194 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493891    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:15:38.500194 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493894    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:15:38.500194 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493896    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:15:38.500194 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493899    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:15:38.500194 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493902    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:15:38.500668 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493904    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:15:38.500668 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493906    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:15:38.500668 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493909    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:15:38.500668 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493912    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:15:38.500668 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493915    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:15:38.500668 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493917    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:15:38.500668 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493920    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:15:38.500668 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493923    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:15:38.500668 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493926    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:15:38.500668 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493929    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:15:38.500668 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493931    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:15:38.500668 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493933    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:15:38.500668 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493936    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:15:38.500668 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493939    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:15:38.500668 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493942    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:15:38.500668 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493944    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:15:38.500668 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493947    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:15:38.500668 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493949    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:15:38.500668 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493952    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:15:38.501124 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493955    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:15:38.501124 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493958    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:15:38.501124 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493961    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:15:38.501124 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493964    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:15:38.501124 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493966    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:15:38.501124 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493969    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:15:38.501124 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493971    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:15:38.501124 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493974    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:15:38.501124 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493976    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:15:38.501124 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493979    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:15:38.501124 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493981    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:15:38.501124 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493986    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:15:38.501124 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493988    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:15:38.501124 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493991    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:15:38.501124 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493993    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:15:38.501124 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493996    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:15:38.501124 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.493999    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:15:38.501124 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494001    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:15:38.501124 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494004    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:15:38.501574 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494007    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:15:38.501574 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494009    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:15:38.501574 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494012    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:15:38.501574 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494014    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:15:38.501574 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494017    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:15:38.501574 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494019    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:15:38.501574 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494022    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:15:38.501574 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494024    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:15:38.501574 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494027    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:15:38.501574 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494029    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:15:38.501574 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494032    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:15:38.501574 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494034    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:15:38.501574 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494037    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:15:38.501574 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494039    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:15:38.501574 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494042    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:15:38.501574 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494045    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:15:38.501574 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494049    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:15:38.501574 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494052    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:15:38.501574 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494055    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:15:38.501574 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494057    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:15:38.502062 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494060    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:15:38.502062 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494063    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:15:38.502062 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494065    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:15:38.502062 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494068    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:15:38.502062 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494071    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:15:38.502062 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494074    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:15:38.502062 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494076    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:15:38.502062 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494080    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:15:38.502062 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494083    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:15:38.502062 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494085    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:15:38.502062 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494087    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:15:38.502062 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494090    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:15:38.502062 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494098    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:15:38.502062 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494102    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:15:38.502062 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.494125    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:15:38.502062 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494157    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:15:38.502441 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494161    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:15:38.502441 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494164    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:15:38.502441 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494167    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:15:38.502441 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494169    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:15:38.502441 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494172    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:15:38.502441 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494174    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:15:38.502441 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494177    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:15:38.502441 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494180    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:15:38.502441 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494182    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:15:38.502441 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494185    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:15:38.502441 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494188    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:15:38.502441 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494190    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:15:38.502441 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494193    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:15:38.502441 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494195    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:15:38.502441 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494198    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:15:38.502441 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494201    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:15:38.502441 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494203    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:15:38.502441 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494206    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:15:38.502441 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494208    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:15:38.502441 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494210    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:15:38.502928 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494214    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:15:38.502928 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494217    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:15:38.502928 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494219    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:15:38.502928 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494222    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:15:38.502928 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494225    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:15:38.502928 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494228    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:15:38.502928 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494230    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:15:38.502928 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494233    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:15:38.502928 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494235    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:15:38.502928 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494238    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:15:38.502928 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494241    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:15:38.502928 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494244    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:15:38.502928 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494246    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:15:38.502928 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494249    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:15:38.502928 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494251    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:15:38.502928 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494254    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:15:38.502928 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494257    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:15:38.502928 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494259    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:15:38.502928 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494262    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:15:38.502928 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494265    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:15:38.503400 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494267    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:15:38.503400 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494270    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:15:38.503400 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494272    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:15:38.503400 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494275    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:15:38.503400 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494278    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:15:38.503400 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494280    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:15:38.503400 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494283    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:15:38.503400 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494285    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:15:38.503400 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494288    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:15:38.503400 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494290    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:15:38.503400 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494293    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:15:38.503400 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494295    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:15:38.503400 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494299    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:15:38.503400 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494301    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:15:38.503400 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494304    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:15:38.503400 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494306    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:15:38.503400 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494310    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:15:38.503400 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494313    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:15:38.503400 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494316    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:15:38.503955 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494319    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:15:38.503955 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494321    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:15:38.503955 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494324    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:15:38.503955 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494326    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:15:38.503955 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494329    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:15:38.503955 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494332    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:15:38.503955 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494336    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:15:38.503955 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494340    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:15:38.503955 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494343    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:15:38.503955 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494346    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:15:38.503955 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494349    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:15:38.503955 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494352    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:15:38.503955 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494355    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:15:38.503955 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494357    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:15:38.503955 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494360    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:15:38.503955 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494363    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:15:38.503955 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494366    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:15:38.503955 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494369    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:15:38.503955 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494371    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:15:38.504410 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494374    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:15:38.504410 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494376    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:15:38.504410 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494379    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:15:38.504410 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494381    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:15:38.504410 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494384    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:15:38.504410 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494386    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:15:38.504410 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494390    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:15:38.504410 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494392    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:15:38.504410 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494395    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:15:38.504410 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494397    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:15:38.504410 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494400    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:15:38.504410 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494403    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:15:38.504410 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494405    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:15:38.504410 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494408    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:15:38.504410 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494411    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:15:38.504410 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494413    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:15:38.504410 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494416    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:15:38.504410 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494419    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:15:38.504410 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494421    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:15:38.504410 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494424    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:15:38.504895 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494426    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:15:38.504895 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494430    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:15:38.504895 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494433    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:15:38.504895 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494436    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:15:38.504895 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494439    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:15:38.504895 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494442    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:15:38.504895 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:38.494444    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:15:38.504895 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.494508    2579 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 19:15:38.504895 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.496478    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 19:15:38.504895 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.498074    2579 server.go:1014] "Starting client certificate rotation"
Oct 30 19:15:38.504895 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.498170    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 19:15:38.504895 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.498214    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 19:15:38.504895 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.500161    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:15:38.504895 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.501272    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:15:38.511445 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.511420    2579 log.go:25] "Validated CRI v1 runtime API"
Oct 30 19:15:38.514401 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.514386    2579 log.go:25] "Validated CRI v1 image API"
Oct 30 19:15:38.515234 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.515211    2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 19:15:38.520674 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.520651    2579 fs.go:135] Filesystem UUIDs: map[543baacd-93ee-48e6-a462-351dd9dea122:/dev/nvme0n1p4 7286354b-3eef-4091-a771-eee36e8f6f81:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Oct 30 19:15:38.520766 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.520675    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 19:15:38.526397 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.526275    2579 manager.go:217] Machine: {Timestamp:2026-10-30 19:15:38.525138898 +0000 UTC m=+0.333319905 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098662 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d309f39a27ba5e9f0e9491ff7e49a SystemUUID:ec2d309f-39a2-7ba5-e9f0-e9491ff7e49a BootID:15fc2b53-f375-489a-af3c-70aed27d98d1 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ef:b9:a9:bb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ef:b9:a9:bb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ba:2e:24:4e:ba:00 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 19:15:38.526397 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.526393    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 19:15:38.526571 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.526555    2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 19:15:38.530089 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.530058    2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 19:15:38.530268 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.530092    2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-84.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 19:15:38.530346 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.530272    2579 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 19:15:38.530346 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.530285    2579 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 19:15:38.530346 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.530319    2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:15:38.532174 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.532162    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:15:38.532896 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.532883    2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 19:15:38.536542 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.536527    2579 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 19:15:38.536680 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.536668    2579 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 19:15:38.537516 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.537505    2579 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 19:15:38.537578 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.537523    2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 19:15:38.537578 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.537538    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 19:15:38.537578 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.537553    2579 kubelet.go:405] "Adding apiserver pod source"
Oct 30 19:15:38.537578 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.537564    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 19:15:38.537981 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.537968    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:15:38.542330 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.542312    2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 19:15:38.546635 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.546621    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:15:38.547492 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.547480    2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 19:15:38.547547 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.547511    2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 19:15:38.547665 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.547656    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 19:15:38.547699 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.547671    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 19:15:38.547699 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.547677    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 19:15:38.547699 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.547682    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 19:15:38.547699 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.547688    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 19:15:38.547699 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.547694    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 19:15:38.548317 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.548308    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 19:15:38.548350 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.548318    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 19:15:38.548350 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.548325    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 19:15:38.548350 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.548331    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 19:15:38.548350 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.548340    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 19:15:38.548350 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.548348    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 19:15:38.548892 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.548883    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 19:15:38.548922 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.548893    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 19:15:38.552719 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.552698    2579 server.go:1263] "Started kubelet"
Oct 30 19:15:38.552822 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.552789    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-84.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 19:15:38.552884 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.552828    2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 19:15:38.552974 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.552918    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 19:15:38.553078 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.552987    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 19:15:38.553594 ip-10-0-137-84 systemd[1]: Started Kubernetes Kubelet.
Oct 30 19:15:38.555154 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.555131    2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 19:15:38.555263 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.555248    2579 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 19:15:38.556930 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.556911    2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 19:15:38.559773 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.559749    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 19:15:38.559910 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:38.559894    2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 19:15:38.560136 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.560120    2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 19:15:38.560325 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:38.560306    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-84.us-east-2.compute.internal\" not found"
Oct 30 19:15:38.560767 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.560747    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 19:15:38.560767 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.560754    2579 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 19:15:38.560901 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.560774    2579 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 19:15:38.560901 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.560880    2579 factory.go:55] Registering systemd factory
Oct 30 19:15:38.560901 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.560897    2579 factory.go:223] Registration of the systemd container factory successfully
Oct 30 19:15:38.561031 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.560882    2579 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 19:15:38.561031 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.560927    2579 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 19:15:38.561130 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.561118    2579 factory.go:153] Registering CRI-O factory
Oct 30 19:15:38.561189 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.561135    2579 factory.go:223] Registration of the crio container factory successfully
Oct 30 19:15:38.561189 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.561182    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 19:15:38.561275 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.561200    2579 factory.go:103] Registering Raw factory
Oct 30 19:15:38.561275 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.561215    2579 manager.go:1199] Started watching for new ooms in manager
Oct 30 19:15:38.562366 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.562352    2579 manager.go:319] Starting recovery of all containers
Oct 30 19:15:38.562432 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:38.562413    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-84.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 19:15:38.562496 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:38.562471    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 19:15:38.562547 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:38.562533    2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-84.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 19:15:38.563495 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:38.562506    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-84.us-east-2.compute.internal.18e364a57cf92fee  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-84.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-84.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 19:15:38.552676334 +0000 UTC m=+0.360857341,LastTimestamp:2026-10-30 19:15:38.552676334 +0000 UTC m=+0.360857341,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-84.us-east-2.compute.internal,}"
Oct 30 19:15:38.573637 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.573617    2579 manager.go:324] Recovery completed
Oct 30 19:15:38.576989 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:38.576963    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 19:15:38.578587 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.578571    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:15:38.580256 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.580232    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-84.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:15:38.580329 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.580269    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-84.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:15:38.580329 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.580279    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-84.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:15:38.580929 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.580916    2579 cpu_manager.go:226] "Starting" policy="none"
Oct 30 19:15:38.580929 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.580926    2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 19:15:38.581017 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.580940    2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 19:15:38.581017 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.580945    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:15:38.581616 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.581601    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-84.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:15:38.581658 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.581636    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-84.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:15:38.581658 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.581650    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-84.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:15:38.582978 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:38.582900    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-84.us-east-2.compute.internal.18e364a57e9e0253  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-84.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-84.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-84.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 19:15:38.580255315 +0000 UTC m=+0.388436322,LastTimestamp:2026-10-30 19:15:38.580255315 +0000 UTC m=+0.388436322,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-84.us-east-2.compute.internal,}"
Oct 30 19:15:38.583120 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.583108    2579 policy_none.go:50] "Start"
Oct 30 19:15:38.583165 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.583126    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 19:15:38.583165 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.583138    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 19:15:38.585143 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.585131    2579 policy_none.go:44] "Start"
Oct 30 19:15:38.592689 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:38.592612    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-84.us-east-2.compute.internal.18e364a57e9e4a36  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-84.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-84.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-84.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 19:15:38.580273718 +0000 UTC m=+0.388454725,LastTimestamp:2026-10-30 19:15:38.580273718 +0000 UTC m=+0.388454725,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-84.us-east-2.compute.internal,}"
Oct 30 19:15:38.599662 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.599645    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sbgdf"
Oct 30 19:15:38.603714 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:38.603654    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-84.us-east-2.compute.internal.18e364a57e9e6e63  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-84.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-137-84.us-east-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-137-84.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 19:15:38.580282979 +0000 UTC m=+0.388463986,LastTimestamp:2026-10-30 19:15:38.580282979 +0000 UTC m=+0.388463986,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-84.us-east-2.compute.internal,}"
Oct 30 19:15:38.611203 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.611183    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sbgdf"
Oct 30 19:15:38.613466 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:38.613271    2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-84.us-east-2.compute.internal.18e364a57e9e0253\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-84.us-east-2.compute.internal.18e364a57e9e0253  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-84.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-84.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-84.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 19:15:38.580255315 +0000 UTC m=+0.388436322,LastTimestamp:2026-10-30 19:15:38.581618279 +0000 UTC m=+0.389799294,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-84.us-east-2.compute.internal,}"
Oct 30 19:15:38.624373 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.624343    2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 19:15:38.651264 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.628435    2579 manager.go:341] "Starting Device Plugin manager"
Oct 30 19:15:38.651264 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:38.628459    2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 19:15:38.651264 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.628468    2579 server.go:87] "Starting device plugin registration server"
Oct 30 19:15:38.651264 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.628714    2579 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 19:15:38.651264 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.628726    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 19:15:38.651264 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.628845    2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 19:15:38.651264 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.628913    2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 19:15:38.651264 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.628921    2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 19:15:38.651264 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:38.629436    2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 19:15:38.651264 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:38.629486    2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-84.us-east-2.compute.internal\" not found"
Oct 30 19:15:38.651264 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.642996    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 19:15:38.651264 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.644314    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 19:15:38.651264 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.644328    2579 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 19:15:38.651264 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.644345    2579 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 19:15:38.651264 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:38.644374    2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 19:15:38.651264 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.647474    2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:15:38.729611 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.729507    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:15:38.731664 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.731644    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-84.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:15:38.731792 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.731678    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-84.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:15:38.731792 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.731692    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-84.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:15:38.731792 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.731715    2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-84.us-east-2.compute.internal"
Oct 30 19:15:38.738167 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.738149    2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-84.us-east-2.compute.internal"
Oct 30 19:15:38.738263 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:38.738177    2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-84.us-east-2.compute.internal\": node \"ip-10-0-137-84.us-east-2.compute.internal\" not found"
Oct 30 19:15:38.749548 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.745103    2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-84.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-84.us-east-2.compute.internal"]
Oct 30 19:15:38.749548 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.745358    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:15:38.750058 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.750042    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-84.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:15:38.750120 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.750074    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-84.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:15:38.750120 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.750083    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-84.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:15:38.751862 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.751848    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:15:38.752004 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.751989    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-84.us-east-2.compute.internal"
Oct 30 19:15:38.752107 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.752029    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:15:38.752623 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.752604    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-84.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:15:38.752686 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.752637    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-84.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:15:38.752686 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.752648    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-84.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:15:38.752686 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.752667    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-84.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:15:38.752686 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.752686    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-84.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:15:38.752853 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.752700    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-84.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:15:38.754387 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.754371    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-84.us-east-2.compute.internal"
Oct 30 19:15:38.754465 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.754401    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:15:38.755108 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.755093    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-84.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:15:38.755182 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.755126    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-84.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:15:38.755182 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.755136    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-84.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:15:38.759351 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:38.759328    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-84.us-east-2.compute.internal\" not found"
Oct 30 19:15:38.761906 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.761890    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2e49edc306f78a76399f3059c085e607-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-84.us-east-2.compute.internal\" (UID: \"2e49edc306f78a76399f3059c085e607\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-84.us-east-2.compute.internal"
Oct 30 19:15:38.761970 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.761914    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e49edc306f78a76399f3059c085e607-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-84.us-east-2.compute.internal\" (UID: \"2e49edc306f78a76399f3059c085e607\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-84.us-east-2.compute.internal"
Oct 30 19:15:38.761970 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.761932    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/532f06e8e868303658c6aeb582644516-config\") pod \"kube-apiserver-proxy-ip-10-0-137-84.us-east-2.compute.internal\" (UID: \"532f06e8e868303658c6aeb582644516\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-84.us-east-2.compute.internal"
Oct 30 19:15:38.778929 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:38.778908    2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-84.us-east-2.compute.internal\" not found" node="ip-10-0-137-84.us-east-2.compute.internal"
Oct 30 19:15:38.781553 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:38.781537    2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-84.us-east-2.compute.internal\" not found" node="ip-10-0-137-84.us-east-2.compute.internal"
Oct 30 19:15:38.859701 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:38.859675    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-84.us-east-2.compute.internal\" not found"
Oct 30 19:15:38.862981 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.862963    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2e49edc306f78a76399f3059c085e607-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-84.us-east-2.compute.internal\" (UID: \"2e49edc306f78a76399f3059c085e607\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-84.us-east-2.compute.internal"
Oct 30 19:15:38.863059 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.862935    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2e49edc306f78a76399f3059c085e607-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-84.us-east-2.compute.internal\" (UID: \"2e49edc306f78a76399f3059c085e607\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-84.us-east-2.compute.internal"
Oct 30 19:15:38.863059 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.863044    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e49edc306f78a76399f3059c085e607-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-84.us-east-2.compute.internal\" (UID: \"2e49edc306f78a76399f3059c085e607\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-84.us-east-2.compute.internal"
Oct 30 19:15:38.863132 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.863062    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/532f06e8e868303658c6aeb582644516-config\") pod \"kube-apiserver-proxy-ip-10-0-137-84.us-east-2.compute.internal\" (UID: \"532f06e8e868303658c6aeb582644516\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-84.us-east-2.compute.internal"
Oct 30 19:15:38.864705 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.863320    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/532f06e8e868303658c6aeb582644516-config\") pod \"kube-apiserver-proxy-ip-10-0-137-84.us-east-2.compute.internal\" (UID: \"532f06e8e868303658c6aeb582644516\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-84.us-east-2.compute.internal"
Oct 30 19:15:38.864705 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:38.863355    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e49edc306f78a76399f3059c085e607-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-84.us-east-2.compute.internal\" (UID: \"2e49edc306f78a76399f3059c085e607\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-84.us-east-2.compute.internal"
Oct 30 19:15:38.959860 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:38.959817    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-84.us-east-2.compute.internal\" not found"
Oct 30 19:15:39.060867 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:39.060804    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-84.us-east-2.compute.internal\" not found"
Oct 30 19:15:39.083105 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:39.083085    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-84.us-east-2.compute.internal"
Oct 30 19:15:39.086019 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:39.086000    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-84.us-east-2.compute.internal"
Oct 30 19:15:39.160904 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:39.160871    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-84.us-east-2.compute.internal\" not found"
Oct 30 19:15:39.261247 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:39.261213    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-84.us-east-2.compute.internal\" not found"
Oct 30 19:15:39.361682 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:39.361615    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-84.us-east-2.compute.internal\" not found"
Oct 30 19:15:39.462075 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:39.462040    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-84.us-east-2.compute.internal\" not found"
Oct 30 19:15:39.497355 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:39.497326    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 19:15:39.497846 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:39.497494    2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 19:15:39.562789 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:39.562764    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-84.us-east-2.compute.internal\" not found"
Oct 30 19:15:39.617389 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:39.617272    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:10:38 +0000 UTC" deadline="2028-03-25 00:12:44.699863463 +0000 UTC"
Oct 30 19:15:39.617389 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:39.617338    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12268h57m5.08252945s"
Oct 30 19:15:39.662977 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:39.662949    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-84.us-east-2.compute.internal\" not found"
Oct 30 19:15:39.687251 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:39.687235    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:15:39.688164 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:39.688148    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-84.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:15:39.688238 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:39.688177    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-84.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:15:39.688238 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:39.688186    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-84.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:15:39.735850 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:39.735817    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e49edc306f78a76399f3059c085e607.slice/crio-d4e2c0ea2c6d38835a0854694a88a86923bd0ddbf7884508c97c088b14da698a WatchSource:0}: Error finding container d4e2c0ea2c6d38835a0854694a88a86923bd0ddbf7884508c97c088b14da698a: Status 404 returned error can't find the container with id d4e2c0ea2c6d38835a0854694a88a86923bd0ddbf7884508c97c088b14da698a
Oct 30 19:15:39.736422 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:39.736400    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod532f06e8e868303658c6aeb582644516.slice/crio-3367f10f4ffc027c5f101ffb7619c5ebe46b3e6ea784512fb62472c7e9c9fd71 WatchSource:0}: Error finding container 3367f10f4ffc027c5f101ffb7619c5ebe46b3e6ea784512fb62472c7e9c9fd71: Status 404 returned error can't find the container with id 3367f10f4ffc027c5f101ffb7619c5ebe46b3e6ea784512fb62472c7e9c9fd71
Oct 30 19:15:39.741544 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:39.741527    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:15:39.763143 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:39.763124    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-84.us-east-2.compute.internal\" not found"
Oct 30 19:15:39.824007 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:39.823984    2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:15:39.864251 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:39.864215    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-84.us-east-2.compute.internal\" not found"
Oct 30 19:15:39.888830 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:39.888781    2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:15:39.951594 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:39.951569    2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:15:39.960959 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:39.960943    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-84.us-east-2.compute.internal"
Oct 30 19:15:39.979399 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:39.979377    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:15:39.980431 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:39.980419    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-84.us-east-2.compute.internal"
Oct 30 19:15:39.989394 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:39.989382    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:15:40.318436 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.318362    2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:15:40.539412 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.539380    2579 apiserver.go:52] "Watching apiserver"
Oct 30 19:15:40.545993 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.545969    2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 19:15:40.546370 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.546341    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-lqmqr","kube-system/kube-apiserver-proxy-ip-10-0-137-84.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-tkdbv","openshift-image-registry/node-ca-zj8d6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-84.us-east-2.compute.internal","openshift-multus/multus-jxqzn","openshift-multus/network-metrics-daemon-n9cws","openshift-network-operator/iptables-alerter-26xq4","openshift-ovn-kubernetes/ovnkube-node-s8t94","openshift-dns/node-resolver-xjjqh","openshift-multus/multus-additional-cni-plugins-w2262","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr","openshift-network-diagnostics/network-check-target-n9zx2","kube-system/global-pull-secret-syncer-wg2df"]
Oct 30 19:15:40.549001 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.548975    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-26xq4"
Oct 30 19:15:40.551165 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.551144    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr"
Oct 30 19:15:40.553072 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.553045    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:15:40.553232 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.553186    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 19:15:40.553232 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.553206    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 19:15:40.553383 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.553255    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xph5z\""
Oct 30 19:15:40.554446 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.554303    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 19:15:40.554446 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.554387    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 19:15:40.554446 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.554404    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7nrdg\""
Oct 30 19:15:40.554602 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.554450    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 19:15:40.555893 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.555874    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:15:40.558551 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.558508    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-w2262"
Oct 30 19:15:40.558840 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.558821    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:15:40.558951 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.558881    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 19:15:40.559058 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.558958    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9tgzd\""
Oct 30 19:15:40.560761 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.560730    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 19:15:40.561113 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.561099    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xjjqh"
Oct 30 19:15:40.562082 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.562064    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jzjv5\""
Oct 30 19:15:40.562295 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.562282    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 19:15:40.562376 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.562359    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 19:15:40.562539 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.562525    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 19:15:40.562759 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.562745    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 19:15:40.562876 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.562858    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 19:15:40.565907 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.565009    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 19:15:40.565907 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.565084    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 19:15:40.565907 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.565140    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-v6xn8\""
Oct 30 19:15:40.566648 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.566624    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.569479 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.569144    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zj8d6"
Oct 30 19:15:40.569479 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.569154    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxkzk\" (UniqueName: \"kubernetes.io/projected/128a8dad-66f8-414b-8c1d-14c7d3b58d25-kube-api-access-jxkzk\") pod \"aws-ebs-csi-driver-node-6k5kr\" (UID: \"128a8dad-66f8-414b-8c1d-14c7d3b58d25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr"
Oct 30 19:15:40.569479 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.569202    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-etc-kubernetes\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:15:40.569479 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.569233    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-etc-sysctl-d\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:15:40.569479 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.569256    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-etc-systemd\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:15:40.569479 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.569313    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/77fbf490-7af3-4bde-8222-4038e2219fff-etc-tuned\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:15:40.569479 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.569362    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dc581227-9542-4571-aced-63d191692a41-ovn-node-metrics-cert\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.569479 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.569402    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5ldg\" (UniqueName: \"kubernetes.io/projected/6331f51e-cf5e-4ba2-9eda-35fd62b14091-kube-api-access-q5ldg\") pod \"iptables-alerter-26xq4\" (UID: \"6331f51e-cf5e-4ba2-9eda-35fd62b14091\") " pod="openshift-network-operator/iptables-alerter-26xq4"
Oct 30 19:15:40.569479 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.569450    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-lib-modules\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:15:40.569479 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.569481    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36ab52a2-65e2-4af6-a872-9052e12714d1-tmp-dir\") pod \"node-resolver-xjjqh\" (UID: \"36ab52a2-65e2-4af6-a872-9052e12714d1\") " pod="openshift-dns/node-resolver-xjjqh"
Oct 30 19:15:40.569970 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.569509    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/128a8dad-66f8-414b-8c1d-14c7d3b58d25-sys-fs\") pod \"aws-ebs-csi-driver-node-6k5kr\" (UID: \"128a8dad-66f8-414b-8c1d-14c7d3b58d25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr"
Oct 30 19:15:40.569970 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.569541    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dda4158a-327b-4b73-8673-3df1c1c0e400-cnibin\") pod \"multus-additional-cni-plugins-w2262\" (UID: \"dda4158a-327b-4b73-8673-3df1c1c0e400\") " pod="openshift-multus/multus-additional-cni-plugins-w2262"
Oct 30 19:15:40.569970 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.569586    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5c96\" (UniqueName: \"kubernetes.io/projected/dda4158a-327b-4b73-8673-3df1c1c0e400-kube-api-access-m5c96\") pod \"multus-additional-cni-plugins-w2262\" (UID: \"dda4158a-327b-4b73-8673-3df1c1c0e400\") " pod="openshift-multus/multus-additional-cni-plugins-w2262"
Oct 30 19:15:40.569970 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.569629    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-node-log\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.569970 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.569664    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dc581227-9542-4571-aced-63d191692a41-env-overrides\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.569970 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.569713    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-run-openvswitch\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.569970 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.569781    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-run-ovn\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.569970 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.569809    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m9lt\" (UniqueName: \"kubernetes.io/projected/dc581227-9542-4571-aced-63d191692a41-kube-api-access-4m9lt\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.569970 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.569860    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77fbf490-7af3-4bde-8222-4038e2219fff-tmp\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:15:40.569970 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.569886    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-var-lib-openvswitch\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.569970 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.569925    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-etc-sysconfig\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:15:40.569970 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.569956    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-sys\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:15:40.570527 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.569994    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/128a8dad-66f8-414b-8c1d-14c7d3b58d25-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6k5kr\" (UID: \"128a8dad-66f8-414b-8c1d-14c7d3b58d25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr"
Oct 30 19:15:40.570527 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.570024    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/36ab52a2-65e2-4af6-a872-9052e12714d1-hosts-file\") pod \"node-resolver-xjjqh\" (UID: \"36ab52a2-65e2-4af6-a872-9052e12714d1\") " pod="openshift-dns/node-resolver-xjjqh"
Oct 30 19:15:40.570527 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.570049    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-host-run-ovn-kubernetes\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.570527 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.570084    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-run\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:15:40.570527 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.570133    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-log-socket\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.570527 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.570160    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-host-slash\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.570527 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.570183    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-host-cni-bin\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.570527 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.570219    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dc581227-9542-4571-aced-63d191692a41-ovnkube-script-lib\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.570527 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.570260    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-host-run-netns\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.570527 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.570297    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/128a8dad-66f8-414b-8c1d-14c7d3b58d25-socket-dir\") pod \"aws-ebs-csi-driver-node-6k5kr\" (UID: \"128a8dad-66f8-414b-8c1d-14c7d3b58d25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr"
Oct 30 19:15:40.570527 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.570323    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/128a8dad-66f8-414b-8c1d-14c7d3b58d25-registration-dir\") pod \"aws-ebs-csi-driver-node-6k5kr\" (UID: \"128a8dad-66f8-414b-8c1d-14c7d3b58d25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr"
Oct 30 19:15:40.570527 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.570365    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-host-kubelet\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.570527 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.570393    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-etc-modprobe-d\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:15:40.570527 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.570435    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dda4158a-327b-4b73-8673-3df1c1c0e400-cni-binary-copy\") pod \"multus-additional-cni-plugins-w2262\" (UID: \"dda4158a-327b-4b73-8673-3df1c1c0e400\") " pod="openshift-multus/multus-additional-cni-plugins-w2262"
Oct 30 19:15:40.570527 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.570461    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dda4158a-327b-4b73-8673-3df1c1c0e400-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w2262\" (UID: \"dda4158a-327b-4b73-8673-3df1c1c0e400\") " pod="openshift-multus/multus-additional-cni-plugins-w2262"
Oct 30 19:15:40.570527 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.570488    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lksw9\" (UniqueName: \"kubernetes.io/projected/36ab52a2-65e2-4af6-a872-9052e12714d1-kube-api-access-lksw9\") pod \"node-resolver-xjjqh\" (UID: \"36ab52a2-65e2-4af6-a872-9052e12714d1\") " pod="openshift-dns/node-resolver-xjjqh"
Oct 30 19:15:40.571360 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.570513    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-systemd-units\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.571360 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.570576    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.571360 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.570603    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6331f51e-cf5e-4ba2-9eda-35fd62b14091-iptables-alerter-script\") pod \"iptables-alerter-26xq4\" (UID: \"6331f51e-cf5e-4ba2-9eda-35fd62b14091\") " pod="openshift-network-operator/iptables-alerter-26xq4"
Oct 30 19:15:40.571360 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.570628    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6331f51e-cf5e-4ba2-9eda-35fd62b14091-host-slash\") pod \"iptables-alerter-26xq4\" (UID: \"6331f51e-cf5e-4ba2-9eda-35fd62b14091\") " pod="openshift-network-operator/iptables-alerter-26xq4"
Oct 30 19:15:40.571360 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.570652    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-var-lib-kubelet\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:15:40.571360 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.570677    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dda4158a-327b-4b73-8673-3df1c1c0e400-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w2262\" (UID: \"dda4158a-327b-4b73-8673-3df1c1c0e400\") " pod="openshift-multus/multus-additional-cni-plugins-w2262"
Oct 30 19:15:40.571360 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.570702    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-etc-sysctl-conf\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:15:40.571360 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.570745    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dda4158a-327b-4b73-8673-3df1c1c0e400-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w2262\" (UID: \"dda4158a-327b-4b73-8673-3df1c1c0e400\") " pod="openshift-multus/multus-additional-cni-plugins-w2262"
Oct 30 19:15:40.571360 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.570789    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-etc-openvswitch\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.571360 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.570852    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/128a8dad-66f8-414b-8c1d-14c7d3b58d25-device-dir\") pod \"aws-ebs-csi-driver-node-6k5kr\" (UID: \"128a8dad-66f8-414b-8c1d-14c7d3b58d25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr"
Oct 30 19:15:40.571360 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.570886    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/128a8dad-66f8-414b-8c1d-14c7d3b58d25-etc-selinux\") pod \"aws-ebs-csi-driver-node-6k5kr\" (UID: \"128a8dad-66f8-414b-8c1d-14c7d3b58d25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr"
Oct 30 19:15:40.571360 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.570909    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-host\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:15:40.571360 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.570968    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8ptq\" (UniqueName: \"kubernetes.io/projected/77fbf490-7af3-4bde-8222-4038e2219fff-kube-api-access-l8ptq\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:15:40.571360 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.571001    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dda4158a-327b-4b73-8673-3df1c1c0e400-system-cni-dir\") pod \"multus-additional-cni-plugins-w2262\" (UID: \"dda4158a-327b-4b73-8673-3df1c1c0e400\") " pod="openshift-multus/multus-additional-cni-plugins-w2262"
Oct 30 19:15:40.571360 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.571053    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-run-systemd\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.571360 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.571094    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dc581227-9542-4571-aced-63d191692a41-ovnkube-config\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.572076 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.571120    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-host-cni-netd\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.572076 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.571439    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9cws"
Oct 30 19:15:40.572076 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:40.571503    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9cws" podUID="e8e297a2-a78a-4665-86d0-6d842bbcaa51"
Oct 30 19:15:40.572368 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.572348    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 19:15:40.573984 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.573967    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.578023 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.577676    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lqmqr"
Oct 30 19:15:40.579559 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.579541    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n9zx2"
Oct 30 19:15:40.579655 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:40.579627    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n9zx2" podUID="47e9832e-bc96-445a-a93e-9d24f84db3b3"
Oct 30 19:15:40.579920 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.579897    2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 19:15:40.582965 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.582940    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wg2df"
Oct 30 19:15:40.583050 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:40.583008    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wg2df" podUID="6ecb643b-1dce-4523-89df-93b22ad8ce6f"
Oct 30 19:15:40.602181 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.602135    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4rzvn"
Oct 30 19:15:40.617046 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.617011    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4rzvn"
Oct 30 19:15:40.649728 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.649672    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-84.us-east-2.compute.internal" event={"ID":"532f06e8e868303658c6aeb582644516","Type":"ContainerStarted","Data":"3367f10f4ffc027c5f101ffb7619c5ebe46b3e6ea784512fb62472c7e9c9fd71"}
Oct 30 19:15:40.651958 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.651918    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-84.us-east-2.compute.internal" event={"ID":"2e49edc306f78a76399f3059c085e607","Type":"ContainerStarted","Data":"d4e2c0ea2c6d38835a0854694a88a86923bd0ddbf7884508c97c088b14da698a"}
Oct 30 19:15:40.658655 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.658557    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 19:15:40.658655 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.658573    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 19:15:40.658655 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.658641    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 19:15:40.658892 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.658660    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 19:15:40.658892 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.658695    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-wctlt\""
Oct 30 19:15:40.658892 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.658701    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 19:15:40.658892 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.658758    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 19:15:40.659643 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.659612    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 19:15:40.659757 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.659685    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-n7s42\""
Oct 30 19:15:40.659757 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.659722    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 19:15:40.659757 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.659723    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 19:15:40.659900 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.659771    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 19:15:40.659900 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.659768    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mglp5\""
Oct 30 19:15:40.659900 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.659779    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 19:15:40.659900 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.659780    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qmpzj\""
Oct 30 19:15:40.661733 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.661718    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 19:15:40.672165 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.672139    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xfxj\" (UniqueName: \"kubernetes.io/projected/e8e297a2-a78a-4665-86d0-6d842bbcaa51-kube-api-access-5xfxj\") pod \"network-metrics-daemon-n9cws\" (UID: \"e8e297a2-a78a-4665-86d0-6d842bbcaa51\") " pod="openshift-multus/network-metrics-daemon-n9cws"
Oct 30 19:15:40.672266 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.672180    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-host-run-k8s-cni-cncf-io\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.672266 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.672205    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-multus-conf-dir\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.672459 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.672267    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/128a8dad-66f8-414b-8c1d-14c7d3b58d25-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6k5kr\" (UID: \"128a8dad-66f8-414b-8c1d-14c7d3b58d25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr"
Oct 30 19:15:40.672459 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.672304    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/36ab52a2-65e2-4af6-a872-9052e12714d1-hosts-file\") pod \"node-resolver-xjjqh\" (UID: \"36ab52a2-65e2-4af6-a872-9052e12714d1\") " pod="openshift-dns/node-resolver-xjjqh"
Oct 30 19:15:40.672459 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.672346    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-host-run-ovn-kubernetes\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.672459 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.672392    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/36ab52a2-65e2-4af6-a872-9052e12714d1-hosts-file\") pod \"node-resolver-xjjqh\" (UID: \"36ab52a2-65e2-4af6-a872-9052e12714d1\") " pod="openshift-dns/node-resolver-xjjqh"
Oct 30 19:15:40.672459 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.672441    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6ecb643b-1dce-4523-89df-93b22ad8ce6f-dbus\") pod \"global-pull-secret-syncer-wg2df\" (UID: \"6ecb643b-1dce-4523-89df-93b22ad8ce6f\") " pod="kube-system/global-pull-secret-syncer-wg2df"
Oct 30 19:15:40.672459 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.672461    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-run\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:15:40.672713 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.672470    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-host-run-ovn-kubernetes\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.672713 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.672503    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-log-socket\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.672713 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.672533    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ecb643b-1dce-4523-89df-93b22ad8ce6f-original-pull-secret\") pod \"global-pull-secret-syncer-wg2df\" (UID: \"6ecb643b-1dce-4523-89df-93b22ad8ce6f\") " pod="kube-system/global-pull-secret-syncer-wg2df"
Oct 30 19:15:40.672713 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.672535    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/128a8dad-66f8-414b-8c1d-14c7d3b58d25-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6k5kr\" (UID: \"128a8dad-66f8-414b-8c1d-14c7d3b58d25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr"
Oct 30 19:15:40.672713 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.672576    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-host-slash\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.672713 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.672619    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-host-cni-bin\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.672713 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.672647    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dc581227-9542-4571-aced-63d191692a41-ovnkube-script-lib\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.672713 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.672675    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-host-var-lib-kubelet\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.672713 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.672676    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-run\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:15:40.673120 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.672731    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-host-run-netns\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.673120 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.672776    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8e297a2-a78a-4665-86d0-6d842bbcaa51-metrics-certs\") pod \"network-metrics-daemon-n9cws\" (UID: \"e8e297a2-a78a-4665-86d0-6d842bbcaa51\") " pod="openshift-multus/network-metrics-daemon-n9cws"
Oct 30 19:15:40.673120 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.672804    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-multus-cni-dir\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.673120 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.672831    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-host-var-lib-cni-multus\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.673120 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.672887    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/128a8dad-66f8-414b-8c1d-14c7d3b58d25-socket-dir\") pod \"aws-ebs-csi-driver-node-6k5kr\" (UID: \"128a8dad-66f8-414b-8c1d-14c7d3b58d25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr"
Oct 30 19:15:40.673120 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.672912    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/128a8dad-66f8-414b-8c1d-14c7d3b58d25-registration-dir\") pod \"aws-ebs-csi-driver-node-6k5kr\" (UID: \"128a8dad-66f8-414b-8c1d-14c7d3b58d25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr"
Oct 30 19:15:40.673120 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.672938    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-host-kubelet\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.673120 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.672965    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gnlf\" (UniqueName: \"kubernetes.io/projected/47e9832e-bc96-445a-a93e-9d24f84db3b3-kube-api-access-6gnlf\") pod \"network-check-target-n9zx2\" (UID: \"47e9832e-bc96-445a-a93e-9d24f84db3b3\") " pod="openshift-network-diagnostics/network-check-target-n9zx2"
Oct 30 19:15:40.673120 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.673018    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbndd\" (UniqueName: \"kubernetes.io/projected/4dbf226e-336f-42c6-a766-cc0ddfe68bef-kube-api-access-lbndd\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.673120 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.673045    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-etc-modprobe-d\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:15:40.673120 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.673070    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dda4158a-327b-4b73-8673-3df1c1c0e400-cni-binary-copy\") pod \"multus-additional-cni-plugins-w2262\" (UID: \"dda4158a-327b-4b73-8673-3df1c1c0e400\") " pod="openshift-multus/multus-additional-cni-plugins-w2262"
Oct 30 19:15:40.673120 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.673096    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dda4158a-327b-4b73-8673-3df1c1c0e400-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w2262\" (UID: \"dda4158a-327b-4b73-8673-3df1c1c0e400\") " pod="openshift-multus/multus-additional-cni-plugins-w2262"
Oct 30 19:15:40.673120 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.673123    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lksw9\" (UniqueName: \"kubernetes.io/projected/36ab52a2-65e2-4af6-a872-9052e12714d1-kube-api-access-lksw9\") pod \"node-resolver-xjjqh\" (UID: \"36ab52a2-65e2-4af6-a872-9052e12714d1\") " pod="openshift-dns/node-resolver-xjjqh"
Oct 30 19:15:40.673548 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.673253    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-systemd-units\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.673548 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.673307    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dc581227-9542-4571-aced-63d191692a41-ovnkube-script-lib\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.673548 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.673320    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.673548 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.673353    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6331f51e-cf5e-4ba2-9eda-35fd62b14091-iptables-alerter-script\") pod \"iptables-alerter-26xq4\" (UID: \"6331f51e-cf5e-4ba2-9eda-35fd62b14091\") " pod="openshift-network-operator/iptables-alerter-26xq4"
Oct 30 19:15:40.673548 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.673404    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6331f51e-cf5e-4ba2-9eda-35fd62b14091-host-slash\") pod \"iptables-alerter-26xq4\" (UID: \"6331f51e-cf5e-4ba2-9eda-35fd62b14091\") " pod="openshift-network-operator/iptables-alerter-26xq4"
Oct 30 19:15:40.673548 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.673427    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-var-lib-kubelet\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:15:40.673548 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.673435    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/128a8dad-66f8-414b-8c1d-14c7d3b58d25-socket-dir\") pod \"aws-ebs-csi-driver-node-6k5kr\" (UID: \"128a8dad-66f8-414b-8c1d-14c7d3b58d25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr"
Oct 30 19:15:40.673548 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.673467    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dda4158a-327b-4b73-8673-3df1c1c0e400-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w2262\" (UID: \"dda4158a-327b-4b73-8673-3df1c1c0e400\") " pod="openshift-multus/multus-additional-cni-plugins-w2262"
Oct 30 19:15:40.673918 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.673551    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-cnibin\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.673918 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.673576    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-etc-kubernetes\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.673918 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.673619    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2d261662-727d-4d68-9f4a-9790ecf16d46-konnectivity-ca\") pod \"konnectivity-agent-lqmqr\" (UID: \"2d261662-727d-4d68-9f4a-9790ecf16d46\") " pod="kube-system/konnectivity-agent-lqmqr"
Oct 30 19:15:40.673918 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.673643    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6ecb643b-1dce-4523-89df-93b22ad8ce6f-kubelet-config\") pod \"global-pull-secret-syncer-wg2df\" (UID: \"6ecb643b-1dce-4523-89df-93b22ad8ce6f\") " pod="kube-system/global-pull-secret-syncer-wg2df"
Oct 30 19:15:40.673918 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.673671    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-etc-sysctl-conf\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:15:40.673918 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.673697    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dda4158a-327b-4b73-8673-3df1c1c0e400-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w2262\" (UID: \"dda4158a-327b-4b73-8673-3df1c1c0e400\") " pod="openshift-multus/multus-additional-cni-plugins-w2262"
Oct 30 19:15:40.673918 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.673726    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-etc-openvswitch\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.673918 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.673735    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-host-slash\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.673918 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.673860    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-var-lib-kubelet\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:15:40.673918 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.673908    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-host-cni-bin\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.674287 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.673948    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-system-cni-dir\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.674287 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.673981    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/128a8dad-66f8-414b-8c1d-14c7d3b58d25-device-dir\") pod \"aws-ebs-csi-driver-node-6k5kr\" (UID: \"128a8dad-66f8-414b-8c1d-14c7d3b58d25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr"
Oct 30 19:15:40.674287 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.674008    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/128a8dad-66f8-414b-8c1d-14c7d3b58d25-etc-selinux\") pod \"aws-ebs-csi-driver-node-6k5kr\" (UID: \"128a8dad-66f8-414b-8c1d-14c7d3b58d25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr"
Oct 30 19:15:40.674287 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.674032    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-host\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:15:40.674287 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.674057    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8ptq\" (UniqueName: \"kubernetes.io/projected/77fbf490-7af3-4bde-8222-4038e2219fff-kube-api-access-l8ptq\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:15:40.674287 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.674081    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dda4158a-327b-4b73-8673-3df1c1c0e400-system-cni-dir\") pod \"multus-additional-cni-plugins-w2262\" (UID: \"dda4158a-327b-4b73-8673-3df1c1c0e400\") " pod="openshift-multus/multus-additional-cni-plugins-w2262"
Oct 30 19:15:40.674287 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.674107    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-run-systemd\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.674287 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.674133    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dc581227-9542-4571-aced-63d191692a41-ovnkube-config\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.674287 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.674157    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-host-cni-netd\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.674287 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.674182    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-host-run-netns\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.674287 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.674207    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2d261662-727d-4d68-9f4a-9790ecf16d46-agent-certs\") pod \"konnectivity-agent-lqmqr\" (UID: \"2d261662-727d-4d68-9f4a-9790ecf16d46\") " pod="kube-system/konnectivity-agent-lqmqr"
Oct 30 19:15:40.674287 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.674234    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxkzk\" (UniqueName: \"kubernetes.io/projected/128a8dad-66f8-414b-8c1d-14c7d3b58d25-kube-api-access-jxkzk\") pod \"aws-ebs-csi-driver-node-6k5kr\" (UID: \"128a8dad-66f8-414b-8c1d-14c7d3b58d25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr"
Oct 30 19:15:40.674287 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.674261    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-etc-kubernetes\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:15:40.674287 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.674286    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-etc-sysctl-d\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:15:40.674799 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.674315    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-etc-systemd\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:15:40.674799 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.674340    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/77fbf490-7af3-4bde-8222-4038e2219fff-etc-tuned\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:15:40.674799 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.674363    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dc581227-9542-4571-aced-63d191692a41-ovn-node-metrics-cert\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.674799 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.674368    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dda4158a-327b-4b73-8673-3df1c1c0e400-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w2262\" (UID: \"dda4158a-327b-4b73-8673-3df1c1c0e400\") " pod="openshift-multus/multus-additional-cni-plugins-w2262"
Oct 30 19:15:40.674799 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.674392    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5ldg\" (UniqueName: \"kubernetes.io/projected/6331f51e-cf5e-4ba2-9eda-35fd62b14091-kube-api-access-q5ldg\") pod \"iptables-alerter-26xq4\" (UID: \"6331f51e-cf5e-4ba2-9eda-35fd62b14091\") " pod="openshift-network-operator/iptables-alerter-26xq4"
Oct 30 19:15:40.674799 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.674418    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-multus-socket-dir-parent\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.674799 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.674419    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-host-run-netns\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.674799 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.674541    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.674799 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.674577    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-lib-modules\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:15:40.674799 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.674603    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36ab52a2-65e2-4af6-a872-9052e12714d1-tmp-dir\") pod \"node-resolver-xjjqh\" (UID: \"36ab52a2-65e2-4af6-a872-9052e12714d1\") " pod="openshift-dns/node-resolver-xjjqh"
Oct 30 19:15:40.674799 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.674631    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-host-var-lib-cni-bin\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.674799 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.674657    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-hostroot\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.674799 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.674695    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/128a8dad-66f8-414b-8c1d-14c7d3b58d25-sys-fs\") pod \"aws-ebs-csi-driver-node-6k5kr\" (UID: \"128a8dad-66f8-414b-8c1d-14c7d3b58d25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr"
Oct 30 19:15:40.674799 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.674729    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dda4158a-327b-4b73-8673-3df1c1c0e400-cnibin\") pod \"multus-additional-cni-plugins-w2262\" (UID: \"dda4158a-327b-4b73-8673-3df1c1c0e400\") " pod="openshift-multus/multus-additional-cni-plugins-w2262"
Oct 30 19:15:40.674799 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.674722    2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 19:15:40.674799 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.674759    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5c96\" (UniqueName: \"kubernetes.io/projected/dda4158a-327b-4b73-8673-3df1c1c0e400-kube-api-access-m5c96\") pod \"multus-additional-cni-plugins-w2262\" (UID: \"dda4158a-327b-4b73-8673-3df1c1c0e400\") " pod="openshift-multus/multus-additional-cni-plugins-w2262"
Oct 30 19:15:40.675467 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.674814    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-node-log\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.675467 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.674839    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dc581227-9542-4571-aced-63d191692a41-env-overrides\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.675467 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.674866    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f1ea27b2-80df-404c-ac39-460f495957b2-serviceca\") pod \"node-ca-zj8d6\" (UID: \"f1ea27b2-80df-404c-ac39-460f495957b2\") " pod="openshift-image-registry/node-ca-zj8d6"
Oct 30 19:15:40.675467 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.674893    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tngcd\" (UniqueName: \"kubernetes.io/projected/f1ea27b2-80df-404c-ac39-460f495957b2-kube-api-access-tngcd\") pod \"node-ca-zj8d6\" (UID: \"f1ea27b2-80df-404c-ac39-460f495957b2\") " pod="openshift-image-registry/node-ca-zj8d6"
Oct 30 19:15:40.675467 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.674919    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-run-openvswitch\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.675467 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.674945    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-run-ovn\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.675467 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.674970    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m9lt\" (UniqueName: \"kubernetes.io/projected/dc581227-9542-4571-aced-63d191692a41-kube-api-access-4m9lt\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.675467 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.674996    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4dbf226e-336f-42c6-a766-cc0ddfe68bef-cni-binary-copy\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.675467 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.675038    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-host-run-multus-certs\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.675467 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.675068    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77fbf490-7af3-4bde-8222-4038e2219fff-tmp\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:15:40.675467 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.675094    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-var-lib-openvswitch\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.675467 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.675129    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4dbf226e-336f-42c6-a766-cc0ddfe68bef-multus-daemon-config\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.675467 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.675170    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-etc-sysconfig\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:15:40.675467 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.675195    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-sys\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:15:40.675467 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.675218    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f1ea27b2-80df-404c-ac39-460f495957b2-host\") pod \"node-ca-zj8d6\" (UID: \"f1ea27b2-80df-404c-ac39-460f495957b2\") " pod="openshift-image-registry/node-ca-zj8d6"
Oct 30 19:15:40.676128 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.675781    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6331f51e-cf5e-4ba2-9eda-35fd62b14091-iptables-alerter-script\") pod \"iptables-alerter-26xq4\" (UID: \"6331f51e-cf5e-4ba2-9eda-35fd62b14091\") " pod="openshift-network-operator/iptables-alerter-26xq4"
Oct 30 19:15:40.676128 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.675906    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-etc-sysctl-conf\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:15:40.676128 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.676027    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dda4158a-327b-4b73-8673-3df1c1c0e400-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w2262\" (UID: \"dda4158a-327b-4b73-8673-3df1c1c0e400\") " pod="openshift-multus/multus-additional-cni-plugins-w2262"
Oct 30 19:15:40.676128 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.676077    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6331f51e-cf5e-4ba2-9eda-35fd62b14091-host-slash\") pod \"iptables-alerter-26xq4\" (UID: \"6331f51e-cf5e-4ba2-9eda-35fd62b14091\") " pod="openshift-network-operator/iptables-alerter-26xq4"
Oct 30 19:15:40.676324 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.676304    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-etc-openvswitch\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.676358 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.676344    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dda4158a-327b-4b73-8673-3df1c1c0e400-system-cni-dir\") pod \"multus-additional-cni-plugins-w2262\" (UID: \"dda4158a-327b-4b73-8673-3df1c1c0e400\") " pod="openshift-multus/multus-additional-cni-plugins-w2262"
Oct 30 19:15:40.676403 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.676393    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/128a8dad-66f8-414b-8c1d-14c7d3b58d25-registration-dir\") pod \"aws-ebs-csi-driver-node-6k5kr\" (UID: \"128a8dad-66f8-414b-8c1d-14c7d3b58d25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr"
Oct 30 19:15:40.676497 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.676426    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-host-kubelet\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.676551 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.676512    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-etc-modprobe-d\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:15:40.676833 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.676796    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-etc-kubernetes\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:15:40.676958 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.676929    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dda4158a-327b-4b73-8673-3df1c1c0e400-cni-binary-copy\") pod \"multus-additional-cni-plugins-w2262\" (UID: \"dda4158a-327b-4b73-8673-3df1c1c0e400\") " pod="openshift-multus/multus-additional-cni-plugins-w2262"
Oct 30 19:15:40.677116 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.677098    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-etc-sysctl-d\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:15:40.677202 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.677153    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-systemd-units\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.677251 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.677199    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-log-socket\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.677251 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.677239    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-run-systemd\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.677367 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.677343    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dda4158a-327b-4b73-8673-3df1c1c0e400-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w2262\" (UID: \"dda4158a-327b-4b73-8673-3df1c1c0e400\") " pod="openshift-multus/multus-additional-cni-plugins-w2262"
Oct 30 19:15:40.677461 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.677416    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/128a8dad-66f8-414b-8c1d-14c7d3b58d25-sys-fs\") pod \"aws-ebs-csi-driver-node-6k5kr\" (UID: \"128a8dad-66f8-414b-8c1d-14c7d3b58d25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr"
Oct 30 19:15:40.677546 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.677459    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dda4158a-327b-4b73-8673-3df1c1c0e400-cnibin\") pod \"multus-additional-cni-plugins-w2262\" (UID: \"dda4158a-327b-4b73-8673-3df1c1c0e400\") " pod="openshift-multus/multus-additional-cni-plugins-w2262"
Oct 30 19:15:40.677714 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.677696    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-node-log\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.677779 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.677766    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dc581227-9542-4571-aced-63d191692a41-ovnkube-config\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.677857 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.677839    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/128a8dad-66f8-414b-8c1d-14c7d3b58d25-device-dir\") pod \"aws-ebs-csi-driver-node-6k5kr\" (UID: \"128a8dad-66f8-414b-8c1d-14c7d3b58d25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr"
Oct 30 19:15:40.677912 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.677883    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-host-cni-netd\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.677966 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.677932    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-host\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:15:40.678013 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.677975    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/128a8dad-66f8-414b-8c1d-14c7d3b58d25-etc-selinux\") pod \"aws-ebs-csi-driver-node-6k5kr\" (UID: \"128a8dad-66f8-414b-8c1d-14c7d3b58d25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr"
Oct 30 19:15:40.678067 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.678048    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-run-openvswitch\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.678133 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.678111    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-run-ovn\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.678222 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.678169    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dc581227-9542-4571-aced-63d191692a41-env-overrides\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.678594 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.678570    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36ab52a2-65e2-4af6-a872-9052e12714d1-tmp-dir\") pod \"node-resolver-xjjqh\" (UID: \"36ab52a2-65e2-4af6-a872-9052e12714d1\") " pod="openshift-dns/node-resolver-xjjqh"
Oct 30 19:15:40.678703 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.678684    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-etc-systemd\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:15:40.679166 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.678813    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-sys\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:15:40.679166 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.678859    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-var-lib-openvswitch\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.679166 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.678913    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-etc-sysconfig\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:15:40.679166 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.678920    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/77fbf490-7af3-4bde-8222-4038e2219fff-etc-tuned\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:15:40.679166 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.679017    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-lib-modules\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:15:40.680814 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.680792    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77fbf490-7af3-4bde-8222-4038e2219fff-tmp\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:15:40.682080 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.682051    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dc581227-9542-4571-aced-63d191692a41-ovn-node-metrics-cert\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.682938 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.682916    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lksw9\" (UniqueName: \"kubernetes.io/projected/36ab52a2-65e2-4af6-a872-9052e12714d1-kube-api-access-lksw9\") pod \"node-resolver-xjjqh\" (UID: \"36ab52a2-65e2-4af6-a872-9052e12714d1\") " pod="openshift-dns/node-resolver-xjjqh"
Oct 30 19:15:40.685093 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.685068    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8ptq\" (UniqueName: \"kubernetes.io/projected/77fbf490-7af3-4bde-8222-4038e2219fff-kube-api-access-l8ptq\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:15:40.685777 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.685730    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxkzk\" (UniqueName: \"kubernetes.io/projected/128a8dad-66f8-414b-8c1d-14c7d3b58d25-kube-api-access-jxkzk\") pod \"aws-ebs-csi-driver-node-6k5kr\" (UID: \"128a8dad-66f8-414b-8c1d-14c7d3b58d25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr"
Oct 30 19:15:40.686944 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.686919    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m9lt\" (UniqueName: \"kubernetes.io/projected/dc581227-9542-4571-aced-63d191692a41-kube-api-access-4m9lt\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.687251 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.687224    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5ldg\" (UniqueName: \"kubernetes.io/projected/6331f51e-cf5e-4ba2-9eda-35fd62b14091-kube-api-access-q5ldg\") pod \"iptables-alerter-26xq4\" (UID: \"6331f51e-cf5e-4ba2-9eda-35fd62b14091\") " pod="openshift-network-operator/iptables-alerter-26xq4"
Oct 30 19:15:40.687504 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.687479    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5c96\" (UniqueName: \"kubernetes.io/projected/dda4158a-327b-4b73-8673-3df1c1c0e400-kube-api-access-m5c96\") pod \"multus-additional-cni-plugins-w2262\" (UID: \"dda4158a-327b-4b73-8673-3df1c1c0e400\") " pod="openshift-multus/multus-additional-cni-plugins-w2262"
Oct 30 19:15:40.775712 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.775677    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-host-run-netns\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.775712 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.775724    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2d261662-727d-4d68-9f4a-9790ecf16d46-agent-certs\") pod \"konnectivity-agent-lqmqr\" (UID: \"2d261662-727d-4d68-9f4a-9790ecf16d46\") " pod="kube-system/konnectivity-agent-lqmqr"
Oct 30 19:15:40.775970 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.775755    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-multus-socket-dir-parent\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.775970 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.775804    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-host-var-lib-cni-bin\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.775970 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.775835    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-hostroot\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.775970 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.775860    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f1ea27b2-80df-404c-ac39-460f495957b2-serviceca\") pod \"node-ca-zj8d6\" (UID: \"f1ea27b2-80df-404c-ac39-460f495957b2\") " pod="openshift-image-registry/node-ca-zj8d6"
Oct 30 19:15:40.775970 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.775881    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tngcd\" (UniqueName: \"kubernetes.io/projected/f1ea27b2-80df-404c-ac39-460f495957b2-kube-api-access-tngcd\") pod \"node-ca-zj8d6\" (UID: \"f1ea27b2-80df-404c-ac39-460f495957b2\") " pod="openshift-image-registry/node-ca-zj8d6"
Oct 30 19:15:40.775970 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.775910    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4dbf226e-336f-42c6-a766-cc0ddfe68bef-cni-binary-copy\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.775970 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.775933    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-host-run-multus-certs\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.775970 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.775960    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4dbf226e-336f-42c6-a766-cc0ddfe68bef-multus-daemon-config\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.776340 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.775985    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f1ea27b2-80df-404c-ac39-460f495957b2-host\") pod \"node-ca-zj8d6\" (UID: \"f1ea27b2-80df-404c-ac39-460f495957b2\") " pod="openshift-image-registry/node-ca-zj8d6"
Oct 30 19:15:40.776340 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.776012    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xfxj\" (UniqueName: \"kubernetes.io/projected/e8e297a2-a78a-4665-86d0-6d842bbcaa51-kube-api-access-5xfxj\") pod \"network-metrics-daemon-n9cws\" (UID: \"e8e297a2-a78a-4665-86d0-6d842bbcaa51\") " pod="openshift-multus/network-metrics-daemon-n9cws"
Oct 30 19:15:40.776340 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.776036    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-host-run-k8s-cni-cncf-io\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.776340 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.776060    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-multus-conf-dir\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.776340 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.776087    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6ecb643b-1dce-4523-89df-93b22ad8ce6f-dbus\") pod \"global-pull-secret-syncer-wg2df\" (UID: \"6ecb643b-1dce-4523-89df-93b22ad8ce6f\") " pod="kube-system/global-pull-secret-syncer-wg2df"
Oct 30 19:15:40.776340 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.776113    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ecb643b-1dce-4523-89df-93b22ad8ce6f-original-pull-secret\") pod \"global-pull-secret-syncer-wg2df\" (UID: \"6ecb643b-1dce-4523-89df-93b22ad8ce6f\") " pod="kube-system/global-pull-secret-syncer-wg2df"
Oct 30 19:15:40.776340 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.776141    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-host-var-lib-kubelet\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.776340 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.776167    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8e297a2-a78a-4665-86d0-6d842bbcaa51-metrics-certs\") pod \"network-metrics-daemon-n9cws\" (UID: \"e8e297a2-a78a-4665-86d0-6d842bbcaa51\") " pod="openshift-multus/network-metrics-daemon-n9cws"
Oct 30 19:15:40.776340 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.776190    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-multus-cni-dir\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.776340 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.776215    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-host-var-lib-cni-multus\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.776340 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.776243    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gnlf\" (UniqueName: \"kubernetes.io/projected/47e9832e-bc96-445a-a93e-9d24f84db3b3-kube-api-access-6gnlf\") pod \"network-check-target-n9zx2\" (UID: \"47e9832e-bc96-445a-a93e-9d24f84db3b3\") " pod="openshift-network-diagnostics/network-check-target-n9zx2"
Oct 30 19:15:40.776340 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.776266    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbndd\" (UniqueName: \"kubernetes.io/projected/4dbf226e-336f-42c6-a766-cc0ddfe68bef-kube-api-access-lbndd\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.776340 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.776297    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-cnibin\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.776340 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.776312    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-etc-kubernetes\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.776340 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.776333    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2d261662-727d-4d68-9f4a-9790ecf16d46-konnectivity-ca\") pod \"konnectivity-agent-lqmqr\" (UID: \"2d261662-727d-4d68-9f4a-9790ecf16d46\") " pod="kube-system/konnectivity-agent-lqmqr"
Oct 30 19:15:40.777021 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.776366    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6ecb643b-1dce-4523-89df-93b22ad8ce6f-kubelet-config\") pod \"global-pull-secret-syncer-wg2df\" (UID: \"6ecb643b-1dce-4523-89df-93b22ad8ce6f\") " pod="kube-system/global-pull-secret-syncer-wg2df"
Oct 30 19:15:40.777021 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.776386    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-system-cni-dir\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.777021 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.776461    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-system-cni-dir\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.777021 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.776491    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-host-run-netns\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.777021 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.776868    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-multus-conf-dir\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.777021 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.776884    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-host-var-lib-cni-multus\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.777021 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.776937    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-multus-socket-dir-parent\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.777021 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.776944    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-host-var-lib-kubelet\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.777021 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.776978    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-host-var-lib-cni-bin\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.777021 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.777018    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-hostroot\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.777409 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.777026    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6ecb643b-1dce-4523-89df-93b22ad8ce6f-dbus\") pod \"global-pull-secret-syncer-wg2df\" (UID: \"6ecb643b-1dce-4523-89df-93b22ad8ce6f\") " pod="kube-system/global-pull-secret-syncer-wg2df"
Oct 30 19:15:40.777409 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:40.777038    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:15:40.777409 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:40.777115    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8e297a2-a78a-4665-86d0-6d842bbcaa51-metrics-certs podName:e8e297a2-a78a-4665-86d0-6d842bbcaa51 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:41.277092095 +0000 UTC m=+3.085273111 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8e297a2-a78a-4665-86d0-6d842bbcaa51-metrics-certs") pod "network-metrics-daemon-n9cws" (UID: "e8e297a2-a78a-4665-86d0-6d842bbcaa51") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:15:40.777409 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.777272    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-multus-cni-dir\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.777409 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.777325    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-cnibin\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.777598 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.777517    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f1ea27b2-80df-404c-ac39-460f495957b2-serviceca\") pod \"node-ca-zj8d6\" (UID: \"f1ea27b2-80df-404c-ac39-460f495957b2\") " pod="openshift-image-registry/node-ca-zj8d6"
Oct 30 19:15:40.777639 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:40.777604    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:15:40.777680 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.777638    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f1ea27b2-80df-404c-ac39-460f495957b2-host\") pod \"node-ca-zj8d6\" (UID: \"f1ea27b2-80df-404c-ac39-460f495957b2\") " pod="openshift-image-registry/node-ca-zj8d6"
Oct 30 19:15:40.777725 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.777680    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-host-run-multus-certs\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.777725 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:40.777701    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ecb643b-1dce-4523-89df-93b22ad8ce6f-original-pull-secret podName:6ecb643b-1dce-4523-89df-93b22ad8ce6f nodeName:}" failed. No retries permitted until 2026-10-30 19:15:41.277681552 +0000 UTC m=+3.085862561 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ecb643b-1dce-4523-89df-93b22ad8ce6f-original-pull-secret") pod "global-pull-secret-syncer-wg2df" (UID: "6ecb643b-1dce-4523-89df-93b22ad8ce6f") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:15:40.777946 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.777896    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-etc-kubernetes\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.777946 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.777896    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-host-run-k8s-cni-cncf-io\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.777946 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.777920    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6ecb643b-1dce-4523-89df-93b22ad8ce6f-kubelet-config\") pod \"global-pull-secret-syncer-wg2df\" (UID: \"6ecb643b-1dce-4523-89df-93b22ad8ce6f\") " pod="kube-system/global-pull-secret-syncer-wg2df"
Oct 30 19:15:40.778725 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.778699    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4dbf226e-336f-42c6-a766-cc0ddfe68bef-cni-binary-copy\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.779016 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.778975    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4dbf226e-336f-42c6-a766-cc0ddfe68bef-multus-daemon-config\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.779134 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.779113    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2d261662-727d-4d68-9f4a-9790ecf16d46-konnectivity-ca\") pod \"konnectivity-agent-lqmqr\" (UID: \"2d261662-727d-4d68-9f4a-9790ecf16d46\") " pod="kube-system/konnectivity-agent-lqmqr"
Oct 30 19:15:40.780607 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.780540    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2d261662-727d-4d68-9f4a-9790ecf16d46-agent-certs\") pod \"konnectivity-agent-lqmqr\" (UID: \"2d261662-727d-4d68-9f4a-9790ecf16d46\") " pod="kube-system/konnectivity-agent-lqmqr"
Oct 30 19:15:40.786145 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:40.786126    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:15:40.786145 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:40.786147    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:15:40.786299 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:40.786157    2579 projected.go:196] Error preparing data for projected volume kube-api-access-6gnlf for pod openshift-network-diagnostics/network-check-target-n9zx2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:15:40.786299 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:40.786211    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47e9832e-bc96-445a-a93e-9d24f84db3b3-kube-api-access-6gnlf podName:47e9832e-bc96-445a-a93e-9d24f84db3b3 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:41.28619537 +0000 UTC m=+3.094376382 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6gnlf" (UniqueName: "kubernetes.io/projected/47e9832e-bc96-445a-a93e-9d24f84db3b3-kube-api-access-6gnlf") pod "network-check-target-n9zx2" (UID: "47e9832e-bc96-445a-a93e-9d24f84db3b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:15:40.788873 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.788850    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tngcd\" (UniqueName: \"kubernetes.io/projected/f1ea27b2-80df-404c-ac39-460f495957b2-kube-api-access-tngcd\") pod \"node-ca-zj8d6\" (UID: \"f1ea27b2-80df-404c-ac39-460f495957b2\") " pod="openshift-image-registry/node-ca-zj8d6"
Oct 30 19:15:40.789468 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.789445    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xfxj\" (UniqueName: \"kubernetes.io/projected/e8e297a2-a78a-4665-86d0-6d842bbcaa51-kube-api-access-5xfxj\") pod \"network-metrics-daemon-n9cws\" (UID: \"e8e297a2-a78a-4665-86d0-6d842bbcaa51\") " pod="openshift-multus/network-metrics-daemon-n9cws"
Oct 30 19:15:40.791894 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.791870    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbndd\" (UniqueName: \"kubernetes.io/projected/4dbf226e-336f-42c6-a766-cc0ddfe68bef-kube-api-access-lbndd\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.866128 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.866044    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-26xq4"
Oct 30 19:15:40.876163 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:40.876112    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6331f51e_cf5e_4ba2_9eda_35fd62b14091.slice/crio-7cf459bbec6deed978519f5d53895993cfcc83c8692fe905e753eb4122488bd9 WatchSource:0}: Error finding container 7cf459bbec6deed978519f5d53895993cfcc83c8692fe905e753eb4122488bd9: Status 404 returned error can't find the container with id 7cf459bbec6deed978519f5d53895993cfcc83c8692fe905e753eb4122488bd9
Oct 30 19:15:40.882515 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.882351    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr"
Oct 30 19:15:40.886619 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.886570    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:15:40.894247 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:40.894213    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod128a8dad_66f8_414b_8c1d_14c7d3b58d25.slice/crio-c7a27c1c553a6b4378a8b33e73553bea17b222b6c46e2fdf059823d42abb7f41 WatchSource:0}: Error finding container c7a27c1c553a6b4378a8b33e73553bea17b222b6c46e2fdf059823d42abb7f41: Status 404 returned error can't find the container with id c7a27c1c553a6b4378a8b33e73553bea17b222b6c46e2fdf059823d42abb7f41
Oct 30 19:15:40.896562 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:40.896531    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77fbf490_7af3_4bde_8222_4038e2219fff.slice/crio-b2fb3c1e6ca634c84d5e914274ea9722133589e4b8f419f17a7baa02ec54c4ff WatchSource:0}: Error finding container b2fb3c1e6ca634c84d5e914274ea9722133589e4b8f419f17a7baa02ec54c4ff: Status 404 returned error can't find the container with id b2fb3c1e6ca634c84d5e914274ea9722133589e4b8f419f17a7baa02ec54c4ff
Oct 30 19:15:40.905627 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.905600    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xjjqh"
Oct 30 19:15:40.907864 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.907839    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-w2262"
Oct 30 19:15:40.918881 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:40.918845    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36ab52a2_65e2_4af6_a872_9052e12714d1.slice/crio-857baf1dd5d42d5472787a331e87333fc9cf66d405389472085f2803f7a4af2c WatchSource:0}: Error finding container 857baf1dd5d42d5472787a331e87333fc9cf66d405389472085f2803f7a4af2c: Status 404 returned error can't find the container with id 857baf1dd5d42d5472787a331e87333fc9cf66d405389472085f2803f7a4af2c
Oct 30 19:15:40.919976 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:40.919953    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddda4158a_327b_4b73_8673_3df1c1c0e400.slice/crio-16a270602f3edfacb90114f8ac0b1d3a49a4c3dd5d021f56175b526b2136d7e2 WatchSource:0}: Error finding container 16a270602f3edfacb90114f8ac0b1d3a49a4c3dd5d021f56175b526b2136d7e2: Status 404 returned error can't find the container with id 16a270602f3edfacb90114f8ac0b1d3a49a4c3dd5d021f56175b526b2136d7e2
Oct 30 19:15:40.920489 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.920473    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zj8d6"
Oct 30 19:15:40.930169 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.930077    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jxqzn"
Oct 30 19:15:40.932941 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.932916    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:15:40.933711 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:40.933601    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1ea27b2_80df_404c_ac39_460f495957b2.slice/crio-eab4e3466e8405b2944cdfca50c0dc8782020e6afc0d3ec51310419af686226e WatchSource:0}: Error finding container eab4e3466e8405b2944cdfca50c0dc8782020e6afc0d3ec51310419af686226e: Status 404 returned error can't find the container with id eab4e3466e8405b2944cdfca50c0dc8782020e6afc0d3ec51310419af686226e
Oct 30 19:15:40.934982 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:40.934964    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lqmqr"
Oct 30 19:15:40.940678 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:40.940554    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dbf226e_336f_42c6_a766_cc0ddfe68bef.slice/crio-ff2ebe84c24d4fd9301608b39c1fd681d19907a9cdea1f6c818f44762d03b0a9 WatchSource:0}: Error finding container ff2ebe84c24d4fd9301608b39c1fd681d19907a9cdea1f6c818f44762d03b0a9: Status 404 returned error can't find the container with id ff2ebe84c24d4fd9301608b39c1fd681d19907a9cdea1f6c818f44762d03b0a9
Oct 30 19:15:40.944995 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:40.944895    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc581227_9542_4571_aced_63d191692a41.slice/crio-bbe001d2613d79b8c9b2749f057a8249b324e97992746982fb7261e20a45458e WatchSource:0}: Error finding container bbe001d2613d79b8c9b2749f057a8249b324e97992746982fb7261e20a45458e: Status 404 returned error can't find the container with id bbe001d2613d79b8c9b2749f057a8249b324e97992746982fb7261e20a45458e
Oct 30 19:15:40.948350 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:15:40.948327    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d261662_727d_4d68_9f4a_9790ecf16d46.slice/crio-087138c34823747582e3393ac259cb9bb6078aad337a07f0de1831b5b7b1c0bf WatchSource:0}: Error finding container 087138c34823747582e3393ac259cb9bb6078aad337a07f0de1831b5b7b1c0bf: Status 404 returned error can't find the container with id 087138c34823747582e3393ac259cb9bb6078aad337a07f0de1831b5b7b1c0bf
Oct 30 19:15:41.280408 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:41.280326    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ecb643b-1dce-4523-89df-93b22ad8ce6f-original-pull-secret\") pod \"global-pull-secret-syncer-wg2df\" (UID: \"6ecb643b-1dce-4523-89df-93b22ad8ce6f\") " pod="kube-system/global-pull-secret-syncer-wg2df"
Oct 30 19:15:41.280408 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:41.280381    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8e297a2-a78a-4665-86d0-6d842bbcaa51-metrics-certs\") pod \"network-metrics-daemon-n9cws\" (UID: \"e8e297a2-a78a-4665-86d0-6d842bbcaa51\") " pod="openshift-multus/network-metrics-daemon-n9cws"
Oct 30 19:15:41.280644 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:41.280512    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:15:41.280644 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:41.280574    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8e297a2-a78a-4665-86d0-6d842bbcaa51-metrics-certs podName:e8e297a2-a78a-4665-86d0-6d842bbcaa51 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:42.280561629 +0000 UTC m=+4.088742624 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8e297a2-a78a-4665-86d0-6d842bbcaa51-metrics-certs") pod "network-metrics-daemon-n9cws" (UID: "e8e297a2-a78a-4665-86d0-6d842bbcaa51") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:15:41.280644 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:41.280617    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:15:41.280746 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:41.280684    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ecb643b-1dce-4523-89df-93b22ad8ce6f-original-pull-secret podName:6ecb643b-1dce-4523-89df-93b22ad8ce6f nodeName:}" failed. No retries permitted until 2026-10-30 19:15:42.280668335 +0000 UTC m=+4.088849343 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ecb643b-1dce-4523-89df-93b22ad8ce6f-original-pull-secret") pod "global-pull-secret-syncer-wg2df" (UID: "6ecb643b-1dce-4523-89df-93b22ad8ce6f") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:15:41.381176 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:41.381145    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gnlf\" (UniqueName: \"kubernetes.io/projected/47e9832e-bc96-445a-a93e-9d24f84db3b3-kube-api-access-6gnlf\") pod \"network-check-target-n9zx2\" (UID: \"47e9832e-bc96-445a-a93e-9d24f84db3b3\") " pod="openshift-network-diagnostics/network-check-target-n9zx2"
Oct 30 19:15:41.381326 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:41.381294    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:15:41.381326 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:41.381308    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:15:41.381326 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:41.381319    2579 projected.go:196] Error preparing data for projected volume kube-api-access-6gnlf for pod openshift-network-diagnostics/network-check-target-n9zx2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:15:41.381439 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:41.381385    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47e9832e-bc96-445a-a93e-9d24f84db3b3-kube-api-access-6gnlf podName:47e9832e-bc96-445a-a93e-9d24f84db3b3 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:42.381365028 +0000 UTC m=+4.189546055 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6gnlf" (UniqueName: "kubernetes.io/projected/47e9832e-bc96-445a-a93e-9d24f84db3b3-kube-api-access-6gnlf") pod "network-check-target-n9zx2" (UID: "47e9832e-bc96-445a-a93e-9d24f84db3b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:15:41.620948 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:41.620625    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:10:40 +0000 UTC" deadline="2028-03-24 21:00:57.190060819 +0000 UTC"
Oct 30 19:15:41.620948 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:41.620904    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12265h45m15.569163593s"
Oct 30 19:15:41.645617 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:41.645585    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9cws"
Oct 30 19:15:41.645799 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:41.645776    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9cws" podUID="e8e297a2-a78a-4665-86d0-6d842bbcaa51"
Oct 30 19:15:41.673296 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:41.673232    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-84.us-east-2.compute.internal" event={"ID":"532f06e8e868303658c6aeb582644516","Type":"ContainerStarted","Data":"2c88a6c99ffe65e37e10acbb712fcdd04dd9da8a7296037eca33248a0774a0e6"}
Oct 30 19:15:41.691890 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:41.691855    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s8t94" event={"ID":"dc581227-9542-4571-aced-63d191692a41","Type":"ContainerStarted","Data":"bbe001d2613d79b8c9b2749f057a8249b324e97992746982fb7261e20a45458e"}
Oct 30 19:15:41.700010 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:41.699943    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jxqzn" event={"ID":"4dbf226e-336f-42c6-a766-cc0ddfe68bef","Type":"ContainerStarted","Data":"ff2ebe84c24d4fd9301608b39c1fd681d19907a9cdea1f6c818f44762d03b0a9"}
Oct 30 19:15:41.707962 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:41.707898    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zj8d6" event={"ID":"f1ea27b2-80df-404c-ac39-460f495957b2","Type":"ContainerStarted","Data":"eab4e3466e8405b2944cdfca50c0dc8782020e6afc0d3ec51310419af686226e"}
Oct 30 19:15:41.715123 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:41.715096    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xjjqh" event={"ID":"36ab52a2-65e2-4af6-a872-9052e12714d1","Type":"ContainerStarted","Data":"857baf1dd5d42d5472787a331e87333fc9cf66d405389472085f2803f7a4af2c"}
Oct 30 19:15:41.717832 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:41.717761    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tkdbv" event={"ID":"77fbf490-7af3-4bde-8222-4038e2219fff","Type":"ContainerStarted","Data":"b2fb3c1e6ca634c84d5e914274ea9722133589e4b8f419f17a7baa02ec54c4ff"}
Oct 30 19:15:41.720119 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:41.720036    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr" event={"ID":"128a8dad-66f8-414b-8c1d-14c7d3b58d25","Type":"ContainerStarted","Data":"c7a27c1c553a6b4378a8b33e73553bea17b222b6c46e2fdf059823d42abb7f41"}
Oct 30 19:15:41.722029 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:41.721966    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lqmqr" event={"ID":"2d261662-727d-4d68-9f4a-9790ecf16d46","Type":"ContainerStarted","Data":"087138c34823747582e3393ac259cb9bb6078aad337a07f0de1831b5b7b1c0bf"}
Oct 30 19:15:41.737353 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:41.737242    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w2262" event={"ID":"dda4158a-327b-4b73-8673-3df1c1c0e400","Type":"ContainerStarted","Data":"16a270602f3edfacb90114f8ac0b1d3a49a4c3dd5d021f56175b526b2136d7e2"}
Oct 30 19:15:41.740534 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:41.740468    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-26xq4" event={"ID":"6331f51e-cf5e-4ba2-9eda-35fd62b14091","Type":"ContainerStarted","Data":"7cf459bbec6deed978519f5d53895993cfcc83c8692fe905e753eb4122488bd9"}
Oct 30 19:15:42.288216 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:42.288172    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ecb643b-1dce-4523-89df-93b22ad8ce6f-original-pull-secret\") pod \"global-pull-secret-syncer-wg2df\" (UID: \"6ecb643b-1dce-4523-89df-93b22ad8ce6f\") " pod="kube-system/global-pull-secret-syncer-wg2df"
Oct 30 19:15:42.288408 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:42.288226    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8e297a2-a78a-4665-86d0-6d842bbcaa51-metrics-certs\") pod \"network-metrics-daemon-n9cws\" (UID: \"e8e297a2-a78a-4665-86d0-6d842bbcaa51\") " pod="openshift-multus/network-metrics-daemon-n9cws"
Oct 30 19:15:42.288408 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:42.288341    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:15:42.288408 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:42.288388    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8e297a2-a78a-4665-86d0-6d842bbcaa51-metrics-certs podName:e8e297a2-a78a-4665-86d0-6d842bbcaa51 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:44.288374905 +0000 UTC m=+6.096555899 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8e297a2-a78a-4665-86d0-6d842bbcaa51-metrics-certs") pod "network-metrics-daemon-n9cws" (UID: "e8e297a2-a78a-4665-86d0-6d842bbcaa51") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:15:42.288816 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:42.288795    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:15:42.288882 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:42.288850    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ecb643b-1dce-4523-89df-93b22ad8ce6f-original-pull-secret podName:6ecb643b-1dce-4523-89df-93b22ad8ce6f nodeName:}" failed. No retries permitted until 2026-10-30 19:15:44.288839176 +0000 UTC m=+6.097020182 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ecb643b-1dce-4523-89df-93b22ad8ce6f-original-pull-secret") pod "global-pull-secret-syncer-wg2df" (UID: "6ecb643b-1dce-4523-89df-93b22ad8ce6f") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:15:42.389167 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:42.389123    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gnlf\" (UniqueName: \"kubernetes.io/projected/47e9832e-bc96-445a-a93e-9d24f84db3b3-kube-api-access-6gnlf\") pod \"network-check-target-n9zx2\" (UID: \"47e9832e-bc96-445a-a93e-9d24f84db3b3\") " pod="openshift-network-diagnostics/network-check-target-n9zx2"
Oct 30 19:15:42.389345 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:42.389309    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:15:42.389345 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:42.389331    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:15:42.389345 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:42.389343    2579 projected.go:196] Error preparing data for projected volume kube-api-access-6gnlf for pod openshift-network-diagnostics/network-check-target-n9zx2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:15:42.389515 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:42.389411    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47e9832e-bc96-445a-a93e-9d24f84db3b3-kube-api-access-6gnlf podName:47e9832e-bc96-445a-a93e-9d24f84db3b3 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:44.389393965 +0000 UTC m=+6.197574965 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6gnlf" (UniqueName: "kubernetes.io/projected/47e9832e-bc96-445a-a93e-9d24f84db3b3-kube-api-access-6gnlf") pod "network-check-target-n9zx2" (UID: "47e9832e-bc96-445a-a93e-9d24f84db3b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:15:42.624800 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:42.621619    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:10:40 +0000 UTC" deadline="2028-04-04 17:00:18.540574439 +0000 UTC"
Oct 30 19:15:42.624800 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:42.621681    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12525h44m35.918897317s"
Oct 30 19:15:42.647103 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:42.647072    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n9zx2"
Oct 30 19:15:42.647277 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:42.647199    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n9zx2" podUID="47e9832e-bc96-445a-a93e-9d24f84db3b3"
Oct 30 19:15:42.647619 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:42.647598    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wg2df"
Oct 30 19:15:42.647767 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:42.647729    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wg2df" podUID="6ecb643b-1dce-4523-89df-93b22ad8ce6f"
Oct 30 19:15:42.750832 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:42.750792    2579 generic.go:356] "Generic (PLEG): container finished" podID="2e49edc306f78a76399f3059c085e607" containerID="ce2eb36c5f6cbceb55fe5ed767c9c6692e1ba7a8da3637235b0de54f5929cf7a" exitCode=0
Oct 30 19:15:42.751617 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:42.751326    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-84.us-east-2.compute.internal" event={"ID":"2e49edc306f78a76399f3059c085e607","Type":"ContainerDied","Data":"ce2eb36c5f6cbceb55fe5ed767c9c6692e1ba7a8da3637235b0de54f5929cf7a"}
Oct 30 19:15:42.766962 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:42.766911    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-84.us-east-2.compute.internal" podStartSLOduration=2.766893847 podStartE2EDuration="2.766893847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:41.689826286 +0000 UTC m=+3.498007300" watchObservedRunningTime="2026-10-30 19:15:42.766893847 +0000 UTC m=+4.575074865"
Oct 30 19:15:43.645108 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:43.644572    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9cws"
Oct 30 19:15:43.645108 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:43.644715    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9cws" podUID="e8e297a2-a78a-4665-86d0-6d842bbcaa51"
Oct 30 19:15:43.774848 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:43.774016    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-84.us-east-2.compute.internal" event={"ID":"2e49edc306f78a76399f3059c085e607","Type":"ContainerStarted","Data":"bf057eaa911bd9bf1028bfb2ba7b4b84535a469b560e362370dec8b7c4c0fa8d"}
Oct 30 19:15:44.307893 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:44.307854    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ecb643b-1dce-4523-89df-93b22ad8ce6f-original-pull-secret\") pod \"global-pull-secret-syncer-wg2df\" (UID: \"6ecb643b-1dce-4523-89df-93b22ad8ce6f\") " pod="kube-system/global-pull-secret-syncer-wg2df"
Oct 30 19:15:44.308118 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:44.308009    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:15:44.308118 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:44.308091    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ecb643b-1dce-4523-89df-93b22ad8ce6f-original-pull-secret podName:6ecb643b-1dce-4523-89df-93b22ad8ce6f nodeName:}" failed. No retries permitted until 2026-10-30 19:15:48.30806783 +0000 UTC m=+10.116248828 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ecb643b-1dce-4523-89df-93b22ad8ce6f-original-pull-secret") pod "global-pull-secret-syncer-wg2df" (UID: "6ecb643b-1dce-4523-89df-93b22ad8ce6f") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:15:44.308118 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:44.308099    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:15:44.308353 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:44.308153    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8e297a2-a78a-4665-86d0-6d842bbcaa51-metrics-certs podName:e8e297a2-a78a-4665-86d0-6d842bbcaa51 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:48.308138669 +0000 UTC m=+10.116319664 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8e297a2-a78a-4665-86d0-6d842bbcaa51-metrics-certs") pod "network-metrics-daemon-n9cws" (UID: "e8e297a2-a78a-4665-86d0-6d842bbcaa51") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:15:44.308353 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:44.308010    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8e297a2-a78a-4665-86d0-6d842bbcaa51-metrics-certs\") pod \"network-metrics-daemon-n9cws\" (UID: \"e8e297a2-a78a-4665-86d0-6d842bbcaa51\") " pod="openshift-multus/network-metrics-daemon-n9cws"
Oct 30 19:15:44.408894 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:44.408845    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gnlf\" (UniqueName: \"kubernetes.io/projected/47e9832e-bc96-445a-a93e-9d24f84db3b3-kube-api-access-6gnlf\") pod \"network-check-target-n9zx2\" (UID: \"47e9832e-bc96-445a-a93e-9d24f84db3b3\") " pod="openshift-network-diagnostics/network-check-target-n9zx2"
Oct 30 19:15:44.409130 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:44.408971    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:15:44.409130 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:44.408986    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:15:44.409130 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:44.408995    2579 projected.go:196] Error preparing data for projected volume kube-api-access-6gnlf for pod openshift-network-diagnostics/network-check-target-n9zx2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:15:44.409130 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:44.409038    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47e9832e-bc96-445a-a93e-9d24f84db3b3-kube-api-access-6gnlf podName:47e9832e-bc96-445a-a93e-9d24f84db3b3 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:48.40902618 +0000 UTC m=+10.217207174 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6gnlf" (UniqueName: "kubernetes.io/projected/47e9832e-bc96-445a-a93e-9d24f84db3b3-kube-api-access-6gnlf") pod "network-check-target-n9zx2" (UID: "47e9832e-bc96-445a-a93e-9d24f84db3b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:15:44.648509 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:44.647753    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n9zx2"
Oct 30 19:15:44.648509 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:44.647878    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n9zx2" podUID="47e9832e-bc96-445a-a93e-9d24f84db3b3"
Oct 30 19:15:44.648509 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:44.648255    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wg2df"
Oct 30 19:15:44.648509 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:44.648352    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wg2df" podUID="6ecb643b-1dce-4523-89df-93b22ad8ce6f"
Oct 30 19:15:44.779448 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:44.779408    2579 generic.go:356] "Generic (PLEG): container finished" podID="dda4158a-327b-4b73-8673-3df1c1c0e400" containerID="9ad16e4217e5ee2917024c4b21645d097d35b4e15156d2a54f77067cd32726e1" exitCode=0
Oct 30 19:15:44.780504 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:44.780474    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w2262" event={"ID":"dda4158a-327b-4b73-8673-3df1c1c0e400","Type":"ContainerDied","Data":"9ad16e4217e5ee2917024c4b21645d097d35b4e15156d2a54f77067cd32726e1"}
Oct 30 19:15:44.806304 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:44.806236    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-84.us-east-2.compute.internal" podStartSLOduration=4.806216471 podStartE2EDuration="4.806216471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:43.790973 +0000 UTC m=+5.599154017" watchObservedRunningTime="2026-10-30 19:15:44.806216471 +0000 UTC m=+6.614397488"
Oct 30 19:15:45.645697 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:45.645213    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9cws"
Oct 30 19:15:45.645697 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:45.645379    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9cws" podUID="e8e297a2-a78a-4665-86d0-6d842bbcaa51"
Oct 30 19:15:45.787969 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:45.787231    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lqmqr" event={"ID":"2d261662-727d-4d68-9f4a-9790ecf16d46","Type":"ContainerStarted","Data":"b9927d6aae7e09573b427ded730f56e02cf016571cd21fd7edff8695b0d8a8fc"}
Oct 30 19:15:46.647600 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:46.647531    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n9zx2"
Oct 30 19:15:46.647718 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:46.647702    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n9zx2" podUID="47e9832e-bc96-445a-a93e-9d24f84db3b3"
Oct 30 19:15:46.647843 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:46.647826    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wg2df"
Oct 30 19:15:46.647933 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:46.647913    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wg2df" podUID="6ecb643b-1dce-4523-89df-93b22ad8ce6f"
Oct 30 19:15:46.795404 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:46.795361    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zj8d6" event={"ID":"f1ea27b2-80df-404c-ac39-460f495957b2","Type":"ContainerStarted","Data":"45d3715ba7dae67b94c994fc781834e66a001e08f3ca7ac7c1f01be135d97e19"}
Oct 30 19:15:46.801520 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:46.801491    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr" event={"ID":"128a8dad-66f8-414b-8c1d-14c7d3b58d25","Type":"ContainerStarted","Data":"e1e1a2c30f314f74bc9c82235f52e2f60ac46ee6d2e28c47a2ef9417a089e3e3"}
Oct 30 19:15:46.812012 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:46.810729    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lqmqr" podStartSLOduration=3.853224731 podStartE2EDuration="7.810711981s" totalImagesPullingTime="3.95748725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:45.803574697 +0000 UTC m=+7.611755715" watchObservedRunningTime="2026-10-30 19:15:46.810711981 +0000 UTC m=+8.618893003"
Oct 30 19:15:47.645098 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:47.645009    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9cws"
Oct 30 19:15:47.645254 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:47.645142    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9cws" podUID="e8e297a2-a78a-4665-86d0-6d842bbcaa51"
Oct 30 19:15:48.275789 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:48.275575    2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 19:15:48.351836 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:48.351050    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ecb643b-1dce-4523-89df-93b22ad8ce6f-original-pull-secret\") pod \"global-pull-secret-syncer-wg2df\" (UID: \"6ecb643b-1dce-4523-89df-93b22ad8ce6f\") " pod="kube-system/global-pull-secret-syncer-wg2df"
Oct 30 19:15:48.351836 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:48.351102    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8e297a2-a78a-4665-86d0-6d842bbcaa51-metrics-certs\") pod \"network-metrics-daemon-n9cws\" (UID: \"e8e297a2-a78a-4665-86d0-6d842bbcaa51\") " pod="openshift-multus/network-metrics-daemon-n9cws"
Oct 30 19:15:48.351836 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:48.351259    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:15:48.351836 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:48.351316    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8e297a2-a78a-4665-86d0-6d842bbcaa51-metrics-certs podName:e8e297a2-a78a-4665-86d0-6d842bbcaa51 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:56.351300534 +0000 UTC m=+18.159481542 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8e297a2-a78a-4665-86d0-6d842bbcaa51-metrics-certs") pod "network-metrics-daemon-n9cws" (UID: "e8e297a2-a78a-4665-86d0-6d842bbcaa51") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:15:48.351836 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:48.351684    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:15:48.351836 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:48.351752    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ecb643b-1dce-4523-89df-93b22ad8ce6f-original-pull-secret podName:6ecb643b-1dce-4523-89df-93b22ad8ce6f nodeName:}" failed. No retries permitted until 2026-10-30 19:15:56.351718244 +0000 UTC m=+18.159899252 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ecb643b-1dce-4523-89df-93b22ad8ce6f-original-pull-secret") pod "global-pull-secret-syncer-wg2df" (UID: "6ecb643b-1dce-4523-89df-93b22ad8ce6f") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:15:48.452381 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:48.452329    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gnlf\" (UniqueName: \"kubernetes.io/projected/47e9832e-bc96-445a-a93e-9d24f84db3b3-kube-api-access-6gnlf\") pod \"network-check-target-n9zx2\" (UID: \"47e9832e-bc96-445a-a93e-9d24f84db3b3\") " pod="openshift-network-diagnostics/network-check-target-n9zx2"
Oct 30 19:15:48.452541 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:48.452511    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:15:48.452541 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:48.452532    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:15:48.452604 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:48.452545    2579 projected.go:196] Error preparing data for projected volume kube-api-access-6gnlf for pod openshift-network-diagnostics/network-check-target-n9zx2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:15:48.452633 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:48.452609    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47e9832e-bc96-445a-a93e-9d24f84db3b3-kube-api-access-6gnlf podName:47e9832e-bc96-445a-a93e-9d24f84db3b3 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:56.452592682 +0000 UTC m=+18.260773682 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6gnlf" (UniqueName: "kubernetes.io/projected/47e9832e-bc96-445a-a93e-9d24f84db3b3-kube-api-access-6gnlf") pod "network-check-target-n9zx2" (UID: "47e9832e-bc96-445a-a93e-9d24f84db3b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:15:48.634557 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:48.634279    2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:15:48.27559761Z","UUID":"51546201-afbf-4821-983b-62da8ebcaaf7","Handler":null,"Name":"","Endpoint":""}
Oct 30 19:15:48.635828 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:48.635800    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 19:15:48.635828 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:48.635831    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 19:15:48.651447 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:48.650337    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n9zx2"
Oct 30 19:15:48.653727 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:48.653706    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wg2df"
Oct 30 19:15:48.653850 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:48.653823    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wg2df" podUID="6ecb643b-1dce-4523-89df-93b22ad8ce6f"
Oct 30 19:15:48.653929 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:48.653907    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n9zx2" podUID="47e9832e-bc96-445a-a93e-9d24f84db3b3"
Oct 30 19:15:48.811334 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:48.811277    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr" event={"ID":"128a8dad-66f8-414b-8c1d-14c7d3b58d25","Type":"ContainerStarted","Data":"2dcb006ca8c4f294fd7d8d07844e55432f22ed4dab636c4692449edfdc9307ea"}
Oct 30 19:15:48.814211 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:48.814160    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jxqzn" event={"ID":"4dbf226e-336f-42c6-a766-cc0ddfe68bef","Type":"ContainerStarted","Data":"1c6eced8a6090fb0fe2ce4b2c92f2f4807a76eb9f3cc9e7941eff0a291878217"}
Oct 30 19:15:48.836207 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:48.835021    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zj8d6" podStartSLOduration=4.632379114 podStartE2EDuration="9.835003506s" totalImagesPullingTime="5.202624392s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:46.811964624 +0000 UTC m=+8.620145643" watchObservedRunningTime="2026-10-30 19:15:48.835003506 +0000 UTC m=+10.643184523"
Oct 30 19:15:48.918394 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:48.917575    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lqmqr"
Oct 30 19:15:48.918943 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:48.918759    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lqmqr"
Oct 30 19:15:48.940266 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:48.939480    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-jxqzn" podStartSLOduration=2.228877858 podStartE2EDuration="9.939460362s" totalImagesPullingTime="7.710582504s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:48.836192372 +0000 UTC m=+10.644373392" watchObservedRunningTime="2026-10-30 19:15:48.939460362 +0000 UTC m=+10.747641382"
Oct 30 19:15:49.646239 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:49.646052    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9cws"
Oct 30 19:15:49.646699 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:49.646337    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9cws" podUID="e8e297a2-a78a-4665-86d0-6d842bbcaa51"
Oct 30 19:15:49.818517 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:49.818426    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xjjqh" event={"ID":"36ab52a2-65e2-4af6-a872-9052e12714d1","Type":"ContainerStarted","Data":"b655c3e46f14932e1c8d7cb96799cae695df29ef53ee47e7703e15b7349fe82a"}
Oct 30 19:15:49.822673 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:49.822582    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr" event={"ID":"128a8dad-66f8-414b-8c1d-14c7d3b58d25","Type":"ContainerStarted","Data":"6cbce1cb7fabac2b279b675dadf0595701da165fc3274b9a7d01fb378c6e02f4"}
Oct 30 19:15:49.826210 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:49.826178    2579 generic.go:356] "Generic (PLEG): container finished" podID="dda4158a-327b-4b73-8673-3df1c1c0e400" containerID="0daaa62fd12d61178eea10740d73a5cf9f347ed5bde484bf1ca2862e3dacea91" exitCode=0
Oct 30 19:15:49.826323 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:49.826265    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w2262" event={"ID":"dda4158a-327b-4b73-8673-3df1c1c0e400","Type":"ContainerDied","Data":"0daaa62fd12d61178eea10740d73a5cf9f347ed5bde484bf1ca2862e3dacea91"}
Oct 30 19:15:49.826712 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:49.826695    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lqmqr"
Oct 30 19:15:49.827030 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:49.826939    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lqmqr"
Oct 30 19:15:49.833791 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:49.833728    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-xjjqh" podStartSLOduration=4.095396739 podStartE2EDuration="11.833710232s" totalImagesPullingTime="7.738313493s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:49.83313791 +0000 UTC m=+11.641318927" watchObservedRunningTime="2026-10-30 19:15:49.833710232 +0000 UTC m=+11.641891249"
Oct 30 19:15:49.886807 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:49.886726    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr" podStartSLOduration=3.764682267 podStartE2EDuration="11.886708301s" totalImagesPullingTime="8.122026034s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:49.86316814 +0000 UTC m=+11.671349158" watchObservedRunningTime="2026-10-30 19:15:49.886708301 +0000 UTC m=+11.694889319"
Oct 30 19:15:50.645020 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:50.644987    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wg2df"
Oct 30 19:15:50.645223 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:50.645140    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wg2df" podUID="6ecb643b-1dce-4523-89df-93b22ad8ce6f"
Oct 30 19:15:50.645317 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:50.645199    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n9zx2"
Oct 30 19:15:50.645317 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:50.645301    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n9zx2" podUID="47e9832e-bc96-445a-a93e-9d24f84db3b3"
Oct 30 19:15:50.830918 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:50.830842    2579 generic.go:356] "Generic (PLEG): container finished" podID="dda4158a-327b-4b73-8673-3df1c1c0e400" containerID="6274a8be63c45b6c33ce5b00999aba3dcf4d72baa2f432e9a4b929b38aefda61" exitCode=0
Oct 30 19:15:50.831361 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:50.830916    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w2262" event={"ID":"dda4158a-327b-4b73-8673-3df1c1c0e400","Type":"ContainerDied","Data":"6274a8be63c45b6c33ce5b00999aba3dcf4d72baa2f432e9a4b929b38aefda61"}
Oct 30 19:15:50.833889 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:50.833865    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-26xq4" event={"ID":"6331f51e-cf5e-4ba2-9eda-35fd62b14091","Type":"ContainerStarted","Data":"cee4f4ab3e3fefa1a75136ae0dffa0d9fb7ec6199492c5074995f8ad2dadc992"}
Oct 30 19:15:50.867866 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:50.867801    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-26xq4" podStartSLOduration=4.668065921 podStartE2EDuration="12.867781871s" totalImagesPullingTime="8.19971595s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:50.867753365 +0000 UTC m=+12.675934384" watchObservedRunningTime="2026-10-30 19:15:50.867781871 +0000 UTC m=+12.675962885"
Oct 30 19:15:51.645623 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:51.645540    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9cws"
Oct 30 19:15:51.645792 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:51.645714    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9cws" podUID="e8e297a2-a78a-4665-86d0-6d842bbcaa51"
Oct 30 19:15:51.837232 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:51.837199    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tkdbv" event={"ID":"77fbf490-7af3-4bde-8222-4038e2219fff","Type":"ContainerStarted","Data":"c2e780d23e81b09fd39a27be276628ed94aa7f61f10b2f458113e550884cf582"}
Oct 30 19:15:51.854372 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:51.854332    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-tkdbv" podStartSLOduration=3.437424535 podStartE2EDuration="13.854318596s" totalImagesPullingTime="10.416894061s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:51.85419804 +0000 UTC m=+13.662379057" watchObservedRunningTime="2026-10-30 19:15:51.854318596 +0000 UTC m=+13.662499611"
Oct 30 19:15:52.644832 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:52.644792    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n9zx2"
Oct 30 19:15:52.644995 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:52.644837    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wg2df"
Oct 30 19:15:52.644995 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:52.644904    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n9zx2" podUID="47e9832e-bc96-445a-a93e-9d24f84db3b3"
Oct 30 19:15:52.645126 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:52.645006    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wg2df" podUID="6ecb643b-1dce-4523-89df-93b22ad8ce6f"
Oct 30 19:15:52.842048 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:52.842015    2579 generic.go:356] "Generic (PLEG): container finished" podID="dda4158a-327b-4b73-8673-3df1c1c0e400" containerID="783623f779ea310dd77c989130e4752363898368076cbcdb419bc0062c39ce81" exitCode=0
Oct 30 19:15:52.842659 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:52.842101    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w2262" event={"ID":"dda4158a-327b-4b73-8673-3df1c1c0e400","Type":"ContainerDied","Data":"783623f779ea310dd77c989130e4752363898368076cbcdb419bc0062c39ce81"}
Oct 30 19:15:53.645643 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:53.645405    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9cws"
Oct 30 19:15:53.645832 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:53.645775    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9cws" podUID="e8e297a2-a78a-4665-86d0-6d842bbcaa51"
Oct 30 19:15:54.645314 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:54.645275    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wg2df"
Oct 30 19:15:54.645778 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:54.645416    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wg2df" podUID="6ecb643b-1dce-4523-89df-93b22ad8ce6f"
Oct 30 19:15:54.645778 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:54.645475    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n9zx2"
Oct 30 19:15:54.645778 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:54.645588    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n9zx2" podUID="47e9832e-bc96-445a-a93e-9d24f84db3b3"
Oct 30 19:15:55.645053 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:55.645036    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9cws"
Oct 30 19:15:55.645232 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:55.645160    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9cws" podUID="e8e297a2-a78a-4665-86d0-6d842bbcaa51"
Oct 30 19:15:55.852114 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:55.851902    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s8t94" event={"ID":"dc581227-9542-4571-aced-63d191692a41","Type":"ContainerStarted","Data":"ef5a2fb97aaa977075b9e6e34866f7446f5ec8fced85397ae4ec5b198da22abc"}
Oct 30 19:15:55.852459 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:55.852172    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s8t94" event={"ID":"dc581227-9542-4571-aced-63d191692a41","Type":"ContainerStarted","Data":"88c260886d8f3c6950980ac113a9464c2da6531227eda8b3d64acf8bf1917964"}
Oct 30 19:15:55.852459 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:55.852214    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s8t94" event={"ID":"dc581227-9542-4571-aced-63d191692a41","Type":"ContainerStarted","Data":"cec7de0fa682395e4a40eff3fb8fbf496c3ad20f0273c9b5e6862b737448283f"}
Oct 30 19:15:55.852459 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:55.852252    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s8t94" event={"ID":"dc581227-9542-4571-aced-63d191692a41","Type":"ContainerStarted","Data":"de3ba135220406029cec3c082962c6c4179b724fd74bf39f5fc474fb68f828df"}
Oct 30 19:15:55.855086 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:55.855060    2579 generic.go:356] "Generic (PLEG): container finished" podID="dda4158a-327b-4b73-8673-3df1c1c0e400" containerID="8a63990b0547bab66217643184c8411c235ef64fe4168470f0c355603bee1d79" exitCode=0
Oct 30 19:15:55.855297 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:55.855102    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w2262" event={"ID":"dda4158a-327b-4b73-8673-3df1c1c0e400","Type":"ContainerDied","Data":"8a63990b0547bab66217643184c8411c235ef64fe4168470f0c355603bee1d79"}
Oct 30 19:15:56.417579 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:56.417546    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ecb643b-1dce-4523-89df-93b22ad8ce6f-original-pull-secret\") pod \"global-pull-secret-syncer-wg2df\" (UID: \"6ecb643b-1dce-4523-89df-93b22ad8ce6f\") " pod="kube-system/global-pull-secret-syncer-wg2df"
Oct 30 19:15:56.417579 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:56.417583    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8e297a2-a78a-4665-86d0-6d842bbcaa51-metrics-certs\") pod \"network-metrics-daemon-n9cws\" (UID: \"e8e297a2-a78a-4665-86d0-6d842bbcaa51\") " pod="openshift-multus/network-metrics-daemon-n9cws"
Oct 30 19:15:56.417881 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:56.417682    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:15:56.417881 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:56.417682    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:15:56.417881 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:56.417765    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8e297a2-a78a-4665-86d0-6d842bbcaa51-metrics-certs podName:e8e297a2-a78a-4665-86d0-6d842bbcaa51 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:12.417726253 +0000 UTC m=+34.225907261 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8e297a2-a78a-4665-86d0-6d842bbcaa51-metrics-certs") pod "network-metrics-daemon-n9cws" (UID: "e8e297a2-a78a-4665-86d0-6d842bbcaa51") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:15:56.417881 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:56.417784    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ecb643b-1dce-4523-89df-93b22ad8ce6f-original-pull-secret podName:6ecb643b-1dce-4523-89df-93b22ad8ce6f nodeName:}" failed. No retries permitted until 2026-10-30 19:16:12.417775292 +0000 UTC m=+34.225956289 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ecb643b-1dce-4523-89df-93b22ad8ce6f-original-pull-secret") pod "global-pull-secret-syncer-wg2df" (UID: "6ecb643b-1dce-4523-89df-93b22ad8ce6f") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:15:56.518746 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:56.518703    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gnlf\" (UniqueName: \"kubernetes.io/projected/47e9832e-bc96-445a-a93e-9d24f84db3b3-kube-api-access-6gnlf\") pod \"network-check-target-n9zx2\" (UID: \"47e9832e-bc96-445a-a93e-9d24f84db3b3\") " pod="openshift-network-diagnostics/network-check-target-n9zx2"
Oct 30 19:15:56.518905 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:56.518823    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:15:56.518905 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:56.518839    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:15:56.518905 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:56.518847    2579 projected.go:196] Error preparing data for projected volume kube-api-access-6gnlf for pod openshift-network-diagnostics/network-check-target-n9zx2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:15:56.518905 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:56.518890    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47e9832e-bc96-445a-a93e-9d24f84db3b3-kube-api-access-6gnlf podName:47e9832e-bc96-445a-a93e-9d24f84db3b3 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:12.518878375 +0000 UTC m=+34.327059369 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6gnlf" (UniqueName: "kubernetes.io/projected/47e9832e-bc96-445a-a93e-9d24f84db3b3-kube-api-access-6gnlf") pod "network-check-target-n9zx2" (UID: "47e9832e-bc96-445a-a93e-9d24f84db3b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:15:56.645547 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:56.645512    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n9zx2"
Oct 30 19:15:56.645547 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:56.645546    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wg2df"
Oct 30 19:15:56.645810 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:56.645616    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n9zx2" podUID="47e9832e-bc96-445a-a93e-9d24f84db3b3"
Oct 30 19:15:56.645810 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:56.645769    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wg2df" podUID="6ecb643b-1dce-4523-89df-93b22ad8ce6f"
Oct 30 19:15:56.861913 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:56.861874    2579 generic.go:356] "Generic (PLEG): container finished" podID="dda4158a-327b-4b73-8673-3df1c1c0e400" containerID="3f3f5c8ef800f8a55e6a657f9c0ed27859a6d16ecde03aa60107aa15a8896659" exitCode=0
Oct 30 19:15:56.862402 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:56.861965    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w2262" event={"ID":"dda4158a-327b-4b73-8673-3df1c1c0e400","Type":"ContainerDied","Data":"3f3f5c8ef800f8a55e6a657f9c0ed27859a6d16ecde03aa60107aa15a8896659"}
Oct 30 19:15:56.865630 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:56.865610    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s8t94" event={"ID":"dc581227-9542-4571-aced-63d191692a41","Type":"ContainerStarted","Data":"a7cfbd21fefd00666389e3f088f7ccb0da256eb20eebb9dee41fe77e5c20b4c5"}
Oct 30 19:15:56.865703 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:56.865670    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s8t94" event={"ID":"dc581227-9542-4571-aced-63d191692a41","Type":"ContainerStarted","Data":"cbaa4c6da4d07ffd974cce0c3e126a2c18907dd999bd08b7fd3d14db101ede3f"}
Oct 30 19:15:57.645198 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:57.645168    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9cws"
Oct 30 19:15:57.645346 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:57.645272    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9cws" podUID="e8e297a2-a78a-4665-86d0-6d842bbcaa51"
Oct 30 19:15:57.871453 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:57.871423    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w2262" event={"ID":"dda4158a-327b-4b73-8673-3df1c1c0e400","Type":"ContainerStarted","Data":"2a2c69cbd9941575bab9bd1338b26286a176b320b0c28b9781508edecfa291ad"}
Oct 30 19:15:57.893058 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:57.893017    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-w2262" podStartSLOduration=8.847234426 podStartE2EDuration="19.893004165s" totalImagesPullingTime="11.045769739s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:38 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:57.892491862 +0000 UTC m=+19.700672878" watchObservedRunningTime="2026-10-30 19:15:57.893004165 +0000 UTC m=+19.701185181"
Oct 30 19:15:58.645978 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:58.645942    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n9zx2"
Oct 30 19:15:58.646175 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:58.646035    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n9zx2" podUID="47e9832e-bc96-445a-a93e-9d24f84db3b3"
Oct 30 19:15:58.646175 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:58.646136    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wg2df"
Oct 30 19:15:58.646285 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:58.646246    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wg2df" podUID="6ecb643b-1dce-4523-89df-93b22ad8ce6f"
Oct 30 19:15:58.876379 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:58.876348    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s8t94" event={"ID":"dc581227-9542-4571-aced-63d191692a41","Type":"ContainerStarted","Data":"a14bc63a81edcd5a38436ce30a3e6c658360705bec926eb76c0a0bea13aaf49d"}
Oct 30 19:15:59.645527 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:15:59.645493    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9cws"
Oct 30 19:15:59.645779 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:15:59.645617    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9cws" podUID="e8e297a2-a78a-4665-86d0-6d842bbcaa51"
Oct 30 19:16:00.644696 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:00.644663    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wg2df"
Oct 30 19:16:00.645305 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:00.644696    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n9zx2"
Oct 30 19:16:00.645305 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:00.644784    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wg2df" podUID="6ecb643b-1dce-4523-89df-93b22ad8ce6f"
Oct 30 19:16:00.645305 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:00.644916    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n9zx2" podUID="47e9832e-bc96-445a-a93e-9d24f84db3b3"
Oct 30 19:16:00.883809 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:00.883764    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s8t94" event={"ID":"dc581227-9542-4571-aced-63d191692a41","Type":"ContainerStarted","Data":"25a50a3786b5c4309101c4455770e519b83643c85c90d2742dad11d578f1db68"}
Oct 30 19:16:00.884055 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:00.884031    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:00.884129 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:00.884094    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:00.900922 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:00.900609    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:00.911886 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:00.911847    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-s8t94" podStartSLOduration=8.25965562 podStartE2EDuration="22.911833939s" totalImagesPullingTime="14.652178319s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:00.910331674 +0000 UTC m=+22.718512691" watchObservedRunningTime="2026-10-30 19:16:00.911833939 +0000 UTC m=+22.720014955"
Oct 30 19:16:01.645366 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:01.645335    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9cws"
Oct 30 19:16:01.645900 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:01.645482    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9cws" podUID="e8e297a2-a78a-4665-86d0-6d842bbcaa51"
Oct 30 19:16:01.854276 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:01.854246    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wg2df"]
Oct 30 19:16:01.854413 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:01.854352    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wg2df"
Oct 30 19:16:01.854473 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:01.854451    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wg2df" podUID="6ecb643b-1dce-4523-89df-93b22ad8ce6f"
Oct 30 19:16:01.857445 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:01.857422    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-n9zx2"]
Oct 30 19:16:01.857586 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:01.857509    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n9zx2"
Oct 30 19:16:01.857648 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:01.857592    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n9zx2" podUID="47e9832e-bc96-445a-a93e-9d24f84db3b3"
Oct 30 19:16:01.858091 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:01.858068    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n9cws"]
Oct 30 19:16:01.885622 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:01.885603    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9cws"
Oct 30 19:16:01.885729 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:01.885688    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9cws" podUID="e8e297a2-a78a-4665-86d0-6d842bbcaa51"
Oct 30 19:16:01.886912 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:01.886887    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:01.902873 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:01.902814    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:03.644674 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:03.644647    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wg2df"
Oct 30 19:16:03.645183 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:03.644684    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n9zx2"
Oct 30 19:16:03.645183 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:03.644770    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9cws"
Oct 30 19:16:03.645183 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:03.644788    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wg2df" podUID="6ecb643b-1dce-4523-89df-93b22ad8ce6f"
Oct 30 19:16:03.645183 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:03.644842    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9cws" podUID="e8e297a2-a78a-4665-86d0-6d842bbcaa51"
Oct 30 19:16:03.645183 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:03.644911    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n9zx2" podUID="47e9832e-bc96-445a-a93e-9d24f84db3b3"
Oct 30 19:16:05.644520 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:05.644491    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n9zx2"
Oct 30 19:16:05.645101 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:05.644491    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wg2df"
Oct 30 19:16:05.645101 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:05.644612    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n9zx2" podUID="47e9832e-bc96-445a-a93e-9d24f84db3b3"
Oct 30 19:16:05.645101 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:05.644498    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9cws"
Oct 30 19:16:05.645101 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:05.644688    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wg2df" podUID="6ecb643b-1dce-4523-89df-93b22ad8ce6f"
Oct 30 19:16:05.645101 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:05.644770    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9cws" podUID="e8e297a2-a78a-4665-86d0-6d842bbcaa51"
Oct 30 19:16:07.645481 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:07.645440    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9cws"
Oct 30 19:16:07.645481 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:07.645475    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wg2df"
Oct 30 19:16:07.645957 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:07.645539    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n9zx2"
Oct 30 19:16:07.645957 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:07.645550    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9cws" podUID="e8e297a2-a78a-4665-86d0-6d842bbcaa51"
Oct 30 19:16:07.645957 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:07.645616    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n9zx2" podUID="47e9832e-bc96-445a-a93e-9d24f84db3b3"
Oct 30 19:16:07.645957 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:07.645667    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wg2df" podUID="6ecb643b-1dce-4523-89df-93b22ad8ce6f"
Oct 30 19:16:08.490345 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.490316    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-84.us-east-2.compute.internal" event="NodeReady"
Oct 30 19:16:08.490508 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.490450    2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 19:16:08.526272 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.526239    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-65kc2"]
Oct 30 19:16:08.529180 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.529160    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6b8sd"]
Oct 30 19:16:08.529542 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.529491    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-65kc2"
Oct 30 19:16:08.532546 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.532527    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-zqvrw"]
Oct 30 19:16:08.532667 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.532651    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6b8sd"
Oct 30 19:16:08.535150 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.535108    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-748f8689f4-6lwjn"]
Oct 30 19:16:08.535422 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.535402    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-zqvrw"
Oct 30 19:16:08.539334 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.539313    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-2sdkw"]
Oct 30 19:16:08.539481 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.539468    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:08.540518 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.540499    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 19:16:08.540608 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.540545    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 19:16:08.542104 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.542087    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-cmw6w"]
Oct 30 19:16:08.542201 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.542187    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2sdkw"
Oct 30 19:16:08.545018 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.544980    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-lpg7w"]
Oct 30 19:16:08.549951 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.549924    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-cmw6w"
Oct 30 19:16:08.551240 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.551217    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-fz9hf"]
Oct 30 19:16:08.551418 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.551397    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-lpg7w"
Oct 30 19:16:08.553595 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.552551    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 19:16:08.553595 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.552593    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 19:16:08.553595 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.552639    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 19:16:08.553595 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.552702    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-bb48c\""
Oct 30 19:16:08.553595 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.553110    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-w9tb6\""
Oct 30 19:16:08.553595 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.553184    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 19:16:08.555538 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.555507    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-75866dc98c-4mnhf"]
Oct 30 19:16:08.556362 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.555703    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-fz9hf"
Oct 30 19:16:08.559646 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.559622    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-62bhw"]
Oct 30 19:16:08.560329 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.559857    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-75866dc98c-4mnhf"
Oct 30 19:16:08.565089 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.564885    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-mg98j"]
Oct 30 19:16:08.566265 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.566199    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-62bhw"
Oct 30 19:16:08.568899 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.568877    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-78dc458d47-tqn8s"]
Oct 30 19:16:08.569065 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.569044    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-mg98j"
Oct 30 19:16:08.572049 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.572032    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-vd8tj"]
Oct 30 19:16:08.572404 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.572381    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:08.575314 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.575293    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p5nzf"]
Oct 30 19:16:08.575445 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.575418    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vd8tj"
Oct 30 19:16:08.578229 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.578207    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-bvcb4"]
Oct 30 19:16:08.578345 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.578328    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p5nzf"
Oct 30 19:16:08.581694 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.581675    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-65kc2"]
Oct 30 19:16:08.581919 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.581872    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6b8sd"]
Oct 30 19:16:08.581919 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.581911    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-zqvrw"]
Oct 30 19:16:08.582121 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.581933    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-2sdkw"]
Oct 30 19:16:08.582121 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.581958    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-lpg7w"]
Oct 30 19:16:08.582121 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.581981    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-mg98j"]
Oct 30 19:16:08.582121 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.582002    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-vd8tj"]
Oct 30 19:16:08.582121 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.582025    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-62bhw"]
Oct 30 19:16:08.582121 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.582047    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gkn6q"]
Oct 30 19:16:08.583168 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.581813    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bvcb4"
Oct 30 19:16:08.588992 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.588970    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9jkcb"]
Oct 30 19:16:08.589532 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.589515    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gkn6q"
Oct 30 19:16:08.592468 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.592446    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-bvcb4"]
Oct 30 19:16:08.592589 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.592496    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p5nzf"]
Oct 30 19:16:08.592589 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.592521    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-fz9hf"]
Oct 30 19:16:08.592589 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.592535    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-75866dc98c-4mnhf"]
Oct 30 19:16:08.592589 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.592563    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-748f8689f4-6lwjn"]
Oct 30 19:16:08.592589 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.592578    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gkn6q"]
Oct 30 19:16:08.592589 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.592578    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9jkcb"
Oct 30 19:16:08.592589 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.592594    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-cmw6w"]
Oct 30 19:16:08.592926 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.592607    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78dc458d47-tqn8s"]
Oct 30 19:16:08.592926 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.592622    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9jkcb"]
Oct 30 19:16:08.596642 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.596617    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndw2r\" (UniqueName: \"kubernetes.io/projected/9201f6f2-73b9-45cd-9950-3a07914d8ea9-kube-api-access-ndw2r\") pod \"kas-connection-checker-685895c7c9-65kc2\" (UID: \"9201f6f2-73b9-45cd-9950-3a07914d8ea9\") " pod="kube-system/kas-connection-checker-685895c7c9-65kc2"
Oct 30 19:16:08.596726 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.596664    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8ww9\" (UniqueName: \"kubernetes.io/projected/00b97df0-857e-4ad9-b33a-4a59777b6f62-kube-api-access-w8ww9\") pod \"kas-connection-checker-685895c7c9-6b8sd\" (UID: \"00b97df0-857e-4ad9-b33a-4a59777b6f62\") " pod="kube-system/kas-connection-checker-685895c7c9-6b8sd"
Oct 30 19:16:08.640213 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.639714    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 19:16:08.640213 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.639986    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:16:08.640610 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.640328    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 19:16:08.640610 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.640413    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 19:16:08.640610 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.640487    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 19:16:08.640610 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.640551    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 19:16:08.640877 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.640658    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 19:16:08.640877 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.640676    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:16:08.640877 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.640829    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 19:16:08.641024 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.640977    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:16:08.641076 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.641060    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 19:16:08.641126 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.641094    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 19:16:08.641177 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.641157    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 19:16:08.641223 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.641201    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:16:08.641268 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.641231    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 19:16:08.641331 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.641278    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 19:16:08.641331 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.641281    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 19:16:08.641483 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.641463    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 19:16:08.641781 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.639998    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 19:16:08.641881 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.641837    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 19:16:08.641881 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.641844    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 19:16:08.641974 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.641917    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 19:16:08.642080 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.641920    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 19:16:08.642486 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.642242    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 19:16:08.642486 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.642354    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:16:08.642486 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.642410    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 19:16:08.642821 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.642710    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 19:16:08.642995 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.642816    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 19:16:08.646949 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.646926    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 19:16:08.651077 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.651051    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 19:16:08.651186 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.651149    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 19:16:08.651186 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.651175    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 19:16:08.651283 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.651244    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 19:16:08.651440 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.651425    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 19:16:08.651786 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.651769    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 19:16:08.651858 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.651833    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-fl6tr\""
Oct 30 19:16:08.651996 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.651980    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 19:16:08.652183 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.652162    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 19:16:08.652272 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.652223    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 19:16:08.652272 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.652233    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wqq7d\""
Oct 30 19:16:08.652272 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.652254    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 19:16:08.652369 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.652279    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 19:16:08.652369 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.652315    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-bfjrl\""
Oct 30 19:16:08.652369 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.652332    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 19:16:08.652476 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.652389    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 19:16:08.652476 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.652405    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lfmhh\""
Oct 30 19:16:08.652476 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.652441    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-c2vmx\""
Oct 30 19:16:08.652654 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.652634    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-fdz8t\""
Oct 30 19:16:08.652707 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.652637    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 19:16:08.652707 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.652696    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-pzpjq\""
Oct 30 19:16:08.652864 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.652800    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-4rpsp\""
Oct 30 19:16:08.652920 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.652864    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-xkqpk\""
Oct 30 19:16:08.652967 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.652949    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-xx4ns\""
Oct 30 19:16:08.653035 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.653024    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-fnphj\""
Oct 30 19:16:08.697777 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.697726    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5002691-8257-4dab-a88b-3c1b60e8e6f8-service-ca-bundle\") pod \"insights-operator-7c4986d449-fz9hf\" (UID: \"e5002691-8257-4dab-a88b-3c1b60e8e6f8\") " pod="openshift-insights/insights-operator-7c4986d449-fz9hf"
Oct 30 19:16:08.697927 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.697804    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/06397fd4-3377-4eaa-8879-ebbb5943fbda-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-2sdkw\" (UID: \"06397fd4-3377-4eaa-8879-ebbb5943fbda\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2sdkw"
Oct 30 19:16:08.697927 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.697827    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-bound-sa-token\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:08.697927 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.697887    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cbc4f7c7-bebd-4200-833e-1afbf3c2c81a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-p5nzf\" (UID: \"cbc4f7c7-bebd-4200-833e-1afbf3c2c81a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p5nzf"
Oct 30 19:16:08.697927 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.697906    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e5002691-8257-4dab-a88b-3c1b60e8e6f8-snapshots\") pod \"insights-operator-7c4986d449-fz9hf\" (UID: \"e5002691-8257-4dab-a88b-3c1b60e8e6f8\") " pod="openshift-insights/insights-operator-7c4986d449-fz9hf"
Oct 30 19:16:08.698054 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.697954    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5002691-8257-4dab-a88b-3c1b60e8e6f8-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-fz9hf\" (UID: \"e5002691-8257-4dab-a88b-3c1b60e8e6f8\") " pod="openshift-insights/insights-operator-7c4986d449-fz9hf"
Oct 30 19:16:08.698054 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.697979    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/76e21916-0744-4499-a0cb-64a7a44a8285-serving-cert\") pod \"console-operator-7cd56b8dbc-cmw6w\" (UID: \"76e21916-0744-4499-a0cb-64a7a44a8285\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cmw6w"
Oct 30 19:16:08.698054 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.697997    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/765d8d9d-b89f-419f-b237-3f8034db1bed-cert\") pod \"ingress-canary-9jkcb\" (UID: \"765d8d9d-b89f-419f-b237-3f8034db1bed\") " pod="openshift-ingress-canary/ingress-canary-9jkcb"
Oct 30 19:16:08.698054 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.698025    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-registry-tls\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:08.698054 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.698040    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a64211d0-1c48-41d2-b254-4ae84d5097d7-ca-trust-extracted\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:08.698193 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.698079    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/247ff425-bed3-4297-b5c0-cc32b21bc179-metrics-certs\") pod \"router-default-75866dc98c-4mnhf\" (UID: \"247ff425-bed3-4297-b5c0-cc32b21bc179\") " pod="openshift-ingress/router-default-75866dc98c-4mnhf"
Oct 30 19:16:08.698193 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.698099    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rz2d\" (UniqueName: \"kubernetes.io/projected/e5002691-8257-4dab-a88b-3c1b60e8e6f8-kube-api-access-9rz2d\") pod \"insights-operator-7c4986d449-fz9hf\" (UID: \"e5002691-8257-4dab-a88b-3c1b60e8e6f8\") " pod="openshift-insights/insights-operator-7c4986d449-fz9hf"
Oct 30 19:16:08.698193 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.698142    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-bound-sa-token\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:08.698193 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.698174    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0269ffa6-ce06-43f3-be2a-58f6aa3e6964-tmp-dir\") pod \"dns-default-gkn6q\" (UID: \"0269ffa6-ce06-43f3-be2a-58f6aa3e6964\") " pod="openshift-dns/dns-default-gkn6q"
Oct 30 19:16:08.698193 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.698190    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/cbc4f7c7-bebd-4200-833e-1afbf3c2c81a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-p5nzf\" (UID: \"cbc4f7c7-bebd-4200-833e-1afbf3c2c81a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p5nzf"
Oct 30 19:16:08.698331 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.698207    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a64211d0-1c48-41d2-b254-4ae84d5097d7-image-registry-private-configuration\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:08.698331 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.698222    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x795\" (UniqueName: \"kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-kube-api-access-6x795\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:08.698331 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.698236    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pfsl\" (UniqueName: \"kubernetes.io/projected/353ebfb4-7e11-41de-b992-277443f56960-kube-api-access-2pfsl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-62bhw\" (UID: \"353ebfb4-7e11-41de-b992-277443f56960\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-62bhw"
Oct 30 19:16:08.698331 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.698280    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a85cca66-a438-4117-9f50-977c7f528b7c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lpg7w\" (UID: \"a85cca66-a438-4117-9f50-977c7f528b7c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lpg7w"
Oct 30 19:16:08.698331 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.698320    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9f302411-01be-438c-a30c-70c944afd630-trusted-ca\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:08.698470 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.698351    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/353ebfb4-7e11-41de-b992-277443f56960-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-62bhw\" (UID: \"353ebfb4-7e11-41de-b992-277443f56960\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-62bhw"
Oct 30 19:16:08.698470 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.698368    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6chk\" (UniqueName: \"kubernetes.io/projected/76e21916-0744-4499-a0cb-64a7a44a8285-kube-api-access-v6chk\") pod \"console-operator-7cd56b8dbc-cmw6w\" (UID: \"76e21916-0744-4499-a0cb-64a7a44a8285\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cmw6w"
Oct 30 19:16:08.698470 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.698394    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a64211d0-1c48-41d2-b254-4ae84d5097d7-registry-certificates\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:08.698470 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.698429    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndw2r\" (UniqueName: \"kubernetes.io/projected/9201f6f2-73b9-45cd-9950-3a07914d8ea9-kube-api-access-ndw2r\") pod \"kas-connection-checker-685895c7c9-65kc2\" (UID: \"9201f6f2-73b9-45cd-9950-3a07914d8ea9\") " pod="kube-system/kas-connection-checker-685895c7c9-65kc2"
Oct 30 19:16:08.698470 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.698459    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0269ffa6-ce06-43f3-be2a-58f6aa3e6964-metrics-tls\") pod \"dns-default-gkn6q\" (UID: \"0269ffa6-ce06-43f3-be2a-58f6aa3e6964\") " pod="openshift-dns/dns-default-gkn6q"
Oct 30 19:16:08.698603 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.698501    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8ww9\" (UniqueName: \"kubernetes.io/projected/00b97df0-857e-4ad9-b33a-4a59777b6f62-kube-api-access-w8ww9\") pod \"kas-connection-checker-685895c7c9-6b8sd\" (UID: \"00b97df0-857e-4ad9-b33a-4a59777b6f62\") " pod="kube-system/kas-connection-checker-685895c7c9-6b8sd"
Oct 30 19:16:08.698603 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.698523    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e5002691-8257-4dab-a88b-3c1b60e8e6f8-serving-cert\") pod \"insights-operator-7c4986d449-fz9hf\" (UID: \"e5002691-8257-4dab-a88b-3c1b60e8e6f8\") " pod="openshift-insights/insights-operator-7c4986d449-fz9hf"
Oct 30 19:16:08.698603 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.698539    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68201777-2ec5-43b2-b587-81e585865d08-serving-cert\") pod \"service-ca-operator-544d749c8c-vd8tj\" (UID: \"68201777-2ec5-43b2-b587-81e585865d08\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vd8tj"
Oct 30 19:16:08.698603 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.698558    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9f302411-01be-438c-a30c-70c944afd630-image-registry-private-configuration\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:08.698603 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.698575    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a64211d0-1c48-41d2-b254-4ae84d5097d7-trusted-ca\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:08.698767 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.698618    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a64211d0-1c48-41d2-b254-4ae84d5097d7-installation-pull-secrets\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:08.698767 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.698646    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a85cca66-a438-4117-9f50-977c7f528b7c-nginx-conf\") pod \"networking-console-plugin-86c478bb88-lpg7w\" (UID: \"a85cca66-a438-4117-9f50-977c7f528b7c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lpg7w"
Oct 30 19:16:08.698767 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.698691    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh979\" (UniqueName: \"kubernetes.io/projected/06397fd4-3377-4eaa-8879-ebbb5943fbda-kube-api-access-zh979\") pod \"cluster-monitoring-operator-884576c57-2sdkw\" (UID: \"06397fd4-3377-4eaa-8879-ebbb5943fbda\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2sdkw"
Oct 30 19:16:08.698767 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.698710    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9f302411-01be-438c-a30c-70c944afd630-installation-pull-secrets\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:08.698767 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.698725    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/06397fd4-3377-4eaa-8879-ebbb5943fbda-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2sdkw\" (UID: \"06397fd4-3377-4eaa-8879-ebbb5943fbda\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2sdkw"
Oct 30 19:16:08.698767 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.698765    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/247ff425-bed3-4297-b5c0-cc32b21bc179-stats-auth\") pod \"router-default-75866dc98c-4mnhf\" (UID: \"247ff425-bed3-4297-b5c0-cc32b21bc179\") " pod="openshift-ingress/router-default-75866dc98c-4mnhf"
Oct 30 19:16:08.698936 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.698786    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsjcc\" (UniqueName: \"kubernetes.io/projected/c84a5c22-7a46-4de1-aef7-32cb9a5453bb-kube-api-access-zsjcc\") pod \"kas-connection-checker-685895c7c9-zqvrw\" (UID: \"c84a5c22-7a46-4de1-aef7-32cb9a5453bb\") " pod="kube-system/kas-connection-checker-685895c7c9-zqvrw"
Oct 30 19:16:08.698936 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.698802    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/68201777-2ec5-43b2-b587-81e585865d08-config\") pod \"service-ca-operator-544d749c8c-vd8tj\" (UID: \"68201777-2ec5-43b2-b587-81e585865d08\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vd8tj"
Oct 30 19:16:08.698936 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.698817    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-registry-tls\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:08.698936 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.698830    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/247ff425-bed3-4297-b5c0-cc32b21bc179-service-ca-bundle\") pod \"router-default-75866dc98c-4mnhf\" (UID: \"247ff425-bed3-4297-b5c0-cc32b21bc179\") " pod="openshift-ingress/router-default-75866dc98c-4mnhf"
Oct 30 19:16:08.698936 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.698859    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28sgd\" (UniqueName: \"kubernetes.io/projected/247ff425-bed3-4297-b5c0-cc32b21bc179-kube-api-access-28sgd\") pod \"router-default-75866dc98c-4mnhf\" (UID: \"247ff425-bed3-4297-b5c0-cc32b21bc179\") " pod="openshift-ingress/router-default-75866dc98c-4mnhf"
Oct 30 19:16:08.698936 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.698900    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/76e21916-0744-4499-a0cb-64a7a44a8285-trusted-ca\") pod \"console-operator-7cd56b8dbc-cmw6w\" (UID: \"76e21916-0744-4499-a0cb-64a7a44a8285\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cmw6w"
Oct 30 19:16:08.698936 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.698928    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/76e21916-0744-4499-a0cb-64a7a44a8285-config\") pod \"console-operator-7cd56b8dbc-cmw6w\" (UID: \"76e21916-0744-4499-a0cb-64a7a44a8285\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cmw6w"
Oct 30 19:16:08.699128 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.698942    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s56kf\" (UniqueName: \"kubernetes.io/projected/68201777-2ec5-43b2-b587-81e585865d08-kube-api-access-s56kf\") pod \"service-ca-operator-544d749c8c-vd8tj\" (UID: \"68201777-2ec5-43b2-b587-81e585865d08\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vd8tj"
Oct 30 19:16:08.699128 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.698960    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpb5t\" (UniqueName: \"kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-kube-api-access-hpb5t\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:08.699128 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.698974    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfglw\" (UniqueName: \"kubernetes.io/projected/2d52a94d-72ac-4e5c-9472-104399a21a3b-kube-api-access-zfglw\") pod \"network-check-source-6f78d59f99-bvcb4\" (UID: \"2d52a94d-72ac-4e5c-9472-104399a21a3b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bvcb4"
Oct 30 19:16:08.699128 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.698990    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9f302411-01be-438c-a30c-70c944afd630-ca-trust-extracted\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:08.699128 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.699021    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/353ebfb4-7e11-41de-b992-277443f56960-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-62bhw\" (UID: \"353ebfb4-7e11-41de-b992-277443f56960\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-62bhw"
Oct 30 19:16:08.699128 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.699051    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6bsd\" (UniqueName: \"kubernetes.io/projected/0269ffa6-ce06-43f3-be2a-58f6aa3e6964-kube-api-access-b6bsd\") pod \"dns-default-gkn6q\" (UID: \"0269ffa6-ce06-43f3-be2a-58f6aa3e6964\") " pod="openshift-dns/dns-default-gkn6q"
Oct 30 19:16:08.699128 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.699071    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/247ff425-bed3-4297-b5c0-cc32b21bc179-default-certificate\") pod \"router-default-75866dc98c-4mnhf\" (UID: \"247ff425-bed3-4297-b5c0-cc32b21bc179\") " pod="openshift-ingress/router-default-75866dc98c-4mnhf"
Oct 30 19:16:08.699128 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.699096    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njpgh\" (UniqueName: \"kubernetes.io/projected/765d8d9d-b89f-419f-b237-3f8034db1bed-kube-api-access-njpgh\") pod \"ingress-canary-9jkcb\" (UID: \"765d8d9d-b89f-419f-b237-3f8034db1bed\") " pod="openshift-ingress-canary/ingress-canary-9jkcb"
Oct 30 19:16:08.699338 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.699135    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tnx8\" (UniqueName: \"kubernetes.io/projected/c243f1d8-1c43-4eb8-93ec-cb28d72cc8a1-kube-api-access-4tnx8\") pod \"volume-data-source-validator-58c9f5f46d-mg98j\" (UID: \"c243f1d8-1c43-4eb8-93ec-cb28d72cc8a1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-mg98j"
Oct 30 19:16:08.699338 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.699173    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9f302411-01be-438c-a30c-70c944afd630-registry-certificates\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:08.699338 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.699194    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0269ffa6-ce06-43f3-be2a-58f6aa3e6964-config-volume\") pod \"dns-default-gkn6q\" (UID: \"0269ffa6-ce06-43f3-be2a-58f6aa3e6964\") " pod="openshift-dns/dns-default-gkn6q"
Oct 30 19:16:08.699338 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.699209    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcgjp\" (UniqueName: \"kubernetes.io/projected/cbc4f7c7-bebd-4200-833e-1afbf3c2c81a-kube-api-access-mcgjp\") pod \"cluster-samples-operator-6ddb6c8b94-p5nzf\" (UID: \"cbc4f7c7-bebd-4200-833e-1afbf3c2c81a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p5nzf"
Oct 30 19:16:08.699338 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.699228    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e5002691-8257-4dab-a88b-3c1b60e8e6f8-tmp\") pod \"insights-operator-7c4986d449-fz9hf\" (UID: \"e5002691-8257-4dab-a88b-3c1b60e8e6f8\") " pod="openshift-insights/insights-operator-7c4986d449-fz9hf"
Oct 30 19:16:08.715804 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.715776    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8ww9\" (UniqueName: \"kubernetes.io/projected/00b97df0-857e-4ad9-b33a-4a59777b6f62-kube-api-access-w8ww9\") pod \"kas-connection-checker-685895c7c9-6b8sd\" (UID: \"00b97df0-857e-4ad9-b33a-4a59777b6f62\") " pod="kube-system/kas-connection-checker-685895c7c9-6b8sd"
Oct 30 19:16:08.717004 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.716979    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndw2r\" (UniqueName: \"kubernetes.io/projected/9201f6f2-73b9-45cd-9950-3a07914d8ea9-kube-api-access-ndw2r\") pod \"kas-connection-checker-685895c7c9-65kc2\" (UID: \"9201f6f2-73b9-45cd-9950-3a07914d8ea9\") " pod="kube-system/kas-connection-checker-685895c7c9-65kc2"
Oct 30 19:16:08.799871 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.799787    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/76e21916-0744-4499-a0cb-64a7a44a8285-config\") pod \"console-operator-7cd56b8dbc-cmw6w\" (UID: \"76e21916-0744-4499-a0cb-64a7a44a8285\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cmw6w"
Oct 30 19:16:08.799871 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.799817    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s56kf\" (UniqueName: \"kubernetes.io/projected/68201777-2ec5-43b2-b587-81e585865d08-kube-api-access-s56kf\") pod \"service-ca-operator-544d749c8c-vd8tj\" (UID: \"68201777-2ec5-43b2-b587-81e585865d08\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vd8tj"
Oct 30 19:16:08.799871 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.799834    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpb5t\" (UniqueName: \"kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-kube-api-access-hpb5t\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:08.799871 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.799850    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfglw\" (UniqueName: \"kubernetes.io/projected/2d52a94d-72ac-4e5c-9472-104399a21a3b-kube-api-access-zfglw\") pod \"network-check-source-6f78d59f99-bvcb4\" (UID: \"2d52a94d-72ac-4e5c-9472-104399a21a3b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bvcb4"
Oct 30 19:16:08.799871 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.799866    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9f302411-01be-438c-a30c-70c944afd630-ca-trust-extracted\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:08.800224 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.799881    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/353ebfb4-7e11-41de-b992-277443f56960-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-62bhw\" (UID: \"353ebfb4-7e11-41de-b992-277443f56960\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-62bhw"
Oct 30 19:16:08.800224 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.800002    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6bsd\" (UniqueName: \"kubernetes.io/projected/0269ffa6-ce06-43f3-be2a-58f6aa3e6964-kube-api-access-b6bsd\") pod \"dns-default-gkn6q\" (UID: \"0269ffa6-ce06-43f3-be2a-58f6aa3e6964\") " pod="openshift-dns/dns-default-gkn6q"
Oct 30 19:16:08.800224 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.800041    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/247ff425-bed3-4297-b5c0-cc32b21bc179-default-certificate\") pod \"router-default-75866dc98c-4mnhf\" (UID: \"247ff425-bed3-4297-b5c0-cc32b21bc179\") " pod="openshift-ingress/router-default-75866dc98c-4mnhf"
Oct 30 19:16:08.800224 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.800070    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njpgh\" (UniqueName: \"kubernetes.io/projected/765d8d9d-b89f-419f-b237-3f8034db1bed-kube-api-access-njpgh\") pod \"ingress-canary-9jkcb\" (UID: \"765d8d9d-b89f-419f-b237-3f8034db1bed\") " pod="openshift-ingress-canary/ingress-canary-9jkcb"
Oct 30 19:16:08.800224 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.800105    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tnx8\" (UniqueName: \"kubernetes.io/projected/c243f1d8-1c43-4eb8-93ec-cb28d72cc8a1-kube-api-access-4tnx8\") pod \"volume-data-source-validator-58c9f5f46d-mg98j\" (UID: \"c243f1d8-1c43-4eb8-93ec-cb28d72cc8a1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-mg98j"
Oct 30 19:16:08.800224 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.800134    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9f302411-01be-438c-a30c-70c944afd630-registry-certificates\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:08.800224 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.800168    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0269ffa6-ce06-43f3-be2a-58f6aa3e6964-config-volume\") pod \"dns-default-gkn6q\" (UID: \"0269ffa6-ce06-43f3-be2a-58f6aa3e6964\") " pod="openshift-dns/dns-default-gkn6q"
Oct 30 19:16:08.800224 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.800194    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcgjp\" (UniqueName: \"kubernetes.io/projected/cbc4f7c7-bebd-4200-833e-1afbf3c2c81a-kube-api-access-mcgjp\") pod \"cluster-samples-operator-6ddb6c8b94-p5nzf\" (UID: \"cbc4f7c7-bebd-4200-833e-1afbf3c2c81a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p5nzf"
Oct 30 19:16:08.800224 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.800219    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e5002691-8257-4dab-a88b-3c1b60e8e6f8-tmp\") pod \"insights-operator-7c4986d449-fz9hf\" (UID: \"e5002691-8257-4dab-a88b-3c1b60e8e6f8\") " pod="openshift-insights/insights-operator-7c4986d449-fz9hf"
Oct 30 19:16:08.800634 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.800246    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5002691-8257-4dab-a88b-3c1b60e8e6f8-service-ca-bundle\") pod \"insights-operator-7c4986d449-fz9hf\" (UID: \"e5002691-8257-4dab-a88b-3c1b60e8e6f8\") " pod="openshift-insights/insights-operator-7c4986d449-fz9hf"
Oct 30 19:16:08.800634 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.800276    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/06397fd4-3377-4eaa-8879-ebbb5943fbda-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-2sdkw\" (UID: \"06397fd4-3377-4eaa-8879-ebbb5943fbda\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2sdkw"
Oct 30 19:16:08.800634 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.800302    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-bound-sa-token\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:08.800634 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.800347    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cbc4f7c7-bebd-4200-833e-1afbf3c2c81a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-p5nzf\" (UID: \"cbc4f7c7-bebd-4200-833e-1afbf3c2c81a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p5nzf"
Oct 30 19:16:08.800634 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.800372    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e5002691-8257-4dab-a88b-3c1b60e8e6f8-snapshots\") pod \"insights-operator-7c4986d449-fz9hf\" (UID: \"e5002691-8257-4dab-a88b-3c1b60e8e6f8\") " pod="openshift-insights/insights-operator-7c4986d449-fz9hf"
Oct 30 19:16:08.800634 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.800625    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9f302411-01be-438c-a30c-70c944afd630-ca-trust-extracted\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:08.800936 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:08.800681    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 19:16:08.800936 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.800695    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/353ebfb4-7e11-41de-b992-277443f56960-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-62bhw\" (UID: \"353ebfb4-7e11-41de-b992-277443f56960\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-62bhw"
Oct 30 19:16:08.800936 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:08.800769    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbc4f7c7-bebd-4200-833e-1afbf3c2c81a-samples-operator-tls podName:cbc4f7c7-bebd-4200-833e-1afbf3c2c81a nodeName:}" failed. No retries permitted until 2026-10-30 19:16:09.300722197 +0000 UTC m=+31.108903205 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cbc4f7c7-bebd-4200-833e-1afbf3c2c81a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-p5nzf" (UID: "cbc4f7c7-bebd-4200-833e-1afbf3c2c81a") : secret "samples-operator-tls" not found
Oct 30 19:16:08.800936 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.800852    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/76e21916-0744-4499-a0cb-64a7a44a8285-config\") pod \"console-operator-7cd56b8dbc-cmw6w\" (UID: \"76e21916-0744-4499-a0cb-64a7a44a8285\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cmw6w"
Oct 30 19:16:08.800936 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.800892    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e5002691-8257-4dab-a88b-3c1b60e8e6f8-tmp\") pod \"insights-operator-7c4986d449-fz9hf\" (UID: \"e5002691-8257-4dab-a88b-3c1b60e8e6f8\") " pod="openshift-insights/insights-operator-7c4986d449-fz9hf"
Oct 30 19:16:08.801175 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.800958    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5002691-8257-4dab-a88b-3c1b60e8e6f8-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-fz9hf\" (UID: \"e5002691-8257-4dab-a88b-3c1b60e8e6f8\") " pod="openshift-insights/insights-operator-7c4986d449-fz9hf"
Oct 30 19:16:08.801175 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.801031    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9f302411-01be-438c-a30c-70c944afd630-registry-certificates\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:08.801175 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.801044    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/76e21916-0744-4499-a0cb-64a7a44a8285-serving-cert\") pod \"console-operator-7cd56b8dbc-cmw6w\" (UID: \"76e21916-0744-4499-a0cb-64a7a44a8285\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cmw6w"
Oct 30 19:16:08.801175 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.801077    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/765d8d9d-b89f-419f-b237-3f8034db1bed-cert\") pod \"ingress-canary-9jkcb\" (UID: \"765d8d9d-b89f-419f-b237-3f8034db1bed\") " pod="openshift-ingress-canary/ingress-canary-9jkcb"
Oct 30 19:16:08.801175 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.801091    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0269ffa6-ce06-43f3-be2a-58f6aa3e6964-config-volume\") pod \"dns-default-gkn6q\" (UID: \"0269ffa6-ce06-43f3-be2a-58f6aa3e6964\") " pod="openshift-dns/dns-default-gkn6q"
Oct 30 19:16:08.801175 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.801118    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-registry-tls\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:08.801461 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.801251    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a64211d0-1c48-41d2-b254-4ae84d5097d7-ca-trust-extracted\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:08.801461 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.801299    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/247ff425-bed3-4297-b5c0-cc32b21bc179-metrics-certs\") pod \"router-default-75866dc98c-4mnhf\" (UID: \"247ff425-bed3-4297-b5c0-cc32b21bc179\") " pod="openshift-ingress/router-default-75866dc98c-4mnhf"
Oct 30 19:16:08.801461 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.801331    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rz2d\" (UniqueName: \"kubernetes.io/projected/e5002691-8257-4dab-a88b-3c1b60e8e6f8-kube-api-access-9rz2d\") pod \"insights-operator-7c4986d449-fz9hf\" (UID: \"e5002691-8257-4dab-a88b-3c1b60e8e6f8\") " pod="openshift-insights/insights-operator-7c4986d449-fz9hf"
Oct 30 19:16:08.801461 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.801361    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-bound-sa-token\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:08.801461 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.801387    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0269ffa6-ce06-43f3-be2a-58f6aa3e6964-tmp-dir\") pod \"dns-default-gkn6q\" (UID: \"0269ffa6-ce06-43f3-be2a-58f6aa3e6964\") " pod="openshift-dns/dns-default-gkn6q"
Oct 30 19:16:08.801461 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.801399    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5002691-8257-4dab-a88b-3c1b60e8e6f8-service-ca-bundle\") pod \"insights-operator-7c4986d449-fz9hf\" (UID: \"e5002691-8257-4dab-a88b-3c1b60e8e6f8\") " pod="openshift-insights/insights-operator-7c4986d449-fz9hf"
Oct 30 19:16:08.801461 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.801407    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/cbc4f7c7-bebd-4200-833e-1afbf3c2c81a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-p5nzf\" (UID: \"cbc4f7c7-bebd-4200-833e-1afbf3c2c81a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p5nzf"
Oct 30 19:16:08.801800 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.801467    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a64211d0-1c48-41d2-b254-4ae84d5097d7-image-registry-private-configuration\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:08.801800 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.801470    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e5002691-8257-4dab-a88b-3c1b60e8e6f8-snapshots\") pod \"insights-operator-7c4986d449-fz9hf\" (UID: \"e5002691-8257-4dab-a88b-3c1b60e8e6f8\") " pod="openshift-insights/insights-operator-7c4986d449-fz9hf"
Oct 30 19:16:08.801800 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:08.801483    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:16:08.801800 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.801496    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x795\" (UniqueName: \"kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-kube-api-access-6x795\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:08.801800 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:08.801530    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/247ff425-bed3-4297-b5c0-cc32b21bc179-metrics-certs podName:247ff425-bed3-4297-b5c0-cc32b21bc179 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:09.301515098 +0000 UTC m=+31.109696105 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/247ff425-bed3-4297-b5c0-cc32b21bc179-metrics-certs") pod "router-default-75866dc98c-4mnhf" (UID: "247ff425-bed3-4297-b5c0-cc32b21bc179") : secret "router-metrics-certs-default" not found
Oct 30 19:16:08.801800 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.801548    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pfsl\" (UniqueName: \"kubernetes.io/projected/353ebfb4-7e11-41de-b992-277443f56960-kube-api-access-2pfsl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-62bhw\" (UID: \"353ebfb4-7e11-41de-b992-277443f56960\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-62bhw"
Oct 30 19:16:08.801800 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.801584    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a85cca66-a438-4117-9f50-977c7f528b7c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lpg7w\" (UID: \"a85cca66-a438-4117-9f50-977c7f528b7c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lpg7w"
Oct 30 19:16:08.801800 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.801615    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9f302411-01be-438c-a30c-70c944afd630-trusted-ca\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:08.802178 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.801915    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0269ffa6-ce06-43f3-be2a-58f6aa3e6964-tmp-dir\") pod \"dns-default-gkn6q\" (UID: \"0269ffa6-ce06-43f3-be2a-58f6aa3e6964\") " pod="openshift-dns/dns-default-gkn6q"
Oct 30 19:16:08.802178 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.801954    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5002691-8257-4dab-a88b-3c1b60e8e6f8-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-fz9hf\" (UID: \"e5002691-8257-4dab-a88b-3c1b60e8e6f8\") " pod="openshift-insights/insights-operator-7c4986d449-fz9hf"
Oct 30 19:16:08.802178 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:08.802048    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:16:08.802178 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:08.802096    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a85cca66-a438-4117-9f50-977c7f528b7c-networking-console-plugin-cert podName:a85cca66-a438-4117-9f50-977c7f528b7c nodeName:}" failed. No retries permitted until 2026-10-30 19:16:09.302080335 +0000 UTC m=+31.110261344 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a85cca66-a438-4117-9f50-977c7f528b7c-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-lpg7w" (UID: "a85cca66-a438-4117-9f50-977c7f528b7c") : secret "networking-console-plugin-cert" not found
Oct 30 19:16:08.802178 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:08.802164    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:16:08.802178 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:08.802176    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-748f8689f4-6lwjn: secret "image-registry-tls" not found
Oct 30 19:16:08.802469 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.802186    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a64211d0-1c48-41d2-b254-4ae84d5097d7-ca-trust-extracted\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:08.802469 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:08.802210    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-registry-tls podName:9f302411-01be-438c-a30c-70c944afd630 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:09.302197669 +0000 UTC m=+31.110378678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-registry-tls") pod "image-registry-748f8689f4-6lwjn" (UID: "9f302411-01be-438c-a30c-70c944afd630") : secret "image-registry-tls" not found
Oct 30 19:16:08.802469 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.802231    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/353ebfb4-7e11-41de-b992-277443f56960-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-62bhw\" (UID: \"353ebfb4-7e11-41de-b992-277443f56960\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-62bhw"
Oct 30 19:16:08.802469 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.802262    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6chk\" (UniqueName: \"kubernetes.io/projected/76e21916-0744-4499-a0cb-64a7a44a8285-kube-api-access-v6chk\") pod \"console-operator-7cd56b8dbc-cmw6w\" (UID: \"76e21916-0744-4499-a0cb-64a7a44a8285\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cmw6w"
Oct 30 19:16:08.802469 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.802298    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a64211d0-1c48-41d2-b254-4ae84d5097d7-registry-certificates\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:08.802469 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.802330    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0269ffa6-ce06-43f3-be2a-58f6aa3e6964-metrics-tls\") pod \"dns-default-gkn6q\" (UID: \"0269ffa6-ce06-43f3-be2a-58f6aa3e6964\") " pod="openshift-dns/dns-default-gkn6q"
Oct 30 19:16:08.802469 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.802357    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e5002691-8257-4dab-a88b-3c1b60e8e6f8-serving-cert\") pod \"insights-operator-7c4986d449-fz9hf\" (UID: \"e5002691-8257-4dab-a88b-3c1b60e8e6f8\") " pod="openshift-insights/insights-operator-7c4986d449-fz9hf"
Oct 30 19:16:08.802469 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.802381    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68201777-2ec5-43b2-b587-81e585865d08-serving-cert\") pod \"service-ca-operator-544d749c8c-vd8tj\" (UID: \"68201777-2ec5-43b2-b587-81e585865d08\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vd8tj"
Oct 30 19:16:08.802469 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.802419    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9f302411-01be-438c-a30c-70c944afd630-image-registry-private-configuration\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:08.802469 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.802471    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a64211d0-1c48-41d2-b254-4ae84d5097d7-trusted-ca\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:08.803022 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.802498    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a64211d0-1c48-41d2-b254-4ae84d5097d7-installation-pull-secrets\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:08.803022 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.802527    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a85cca66-a438-4117-9f50-977c7f528b7c-nginx-conf\") pod \"networking-console-plugin-86c478bb88-lpg7w\" (UID: \"a85cca66-a438-4117-9f50-977c7f528b7c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lpg7w"
Oct 30 19:16:08.803022 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.802546    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/cbc4f7c7-bebd-4200-833e-1afbf3c2c81a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-p5nzf\" (UID: \"cbc4f7c7-bebd-4200-833e-1afbf3c2c81a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p5nzf"
Oct 30 19:16:08.803022 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.802577    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh979\" (UniqueName: \"kubernetes.io/projected/06397fd4-3377-4eaa-8879-ebbb5943fbda-kube-api-access-zh979\") pod \"cluster-monitoring-operator-884576c57-2sdkw\" (UID: \"06397fd4-3377-4eaa-8879-ebbb5943fbda\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2sdkw"
Oct 30 19:16:08.803022 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.802605    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9f302411-01be-438c-a30c-70c944afd630-installation-pull-secrets\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:08.803022 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.802632    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/06397fd4-3377-4eaa-8879-ebbb5943fbda-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2sdkw\" (UID: \"06397fd4-3377-4eaa-8879-ebbb5943fbda\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2sdkw"
Oct 30 19:16:08.803022 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.802658    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/247ff425-bed3-4297-b5c0-cc32b21bc179-stats-auth\") pod \"router-default-75866dc98c-4mnhf\" (UID: \"247ff425-bed3-4297-b5c0-cc32b21bc179\") " pod="openshift-ingress/router-default-75866dc98c-4mnhf"
Oct 30 19:16:08.803022 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.802687    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsjcc\" (UniqueName: \"kubernetes.io/projected/c84a5c22-7a46-4de1-aef7-32cb9a5453bb-kube-api-access-zsjcc\") pod \"kas-connection-checker-685895c7c9-zqvrw\" (UID: \"c84a5c22-7a46-4de1-aef7-32cb9a5453bb\") " pod="kube-system/kas-connection-checker-685895c7c9-zqvrw"
Oct 30 19:16:08.803022 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.802712    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/68201777-2ec5-43b2-b587-81e585865d08-config\") pod \"service-ca-operator-544d749c8c-vd8tj\" (UID: \"68201777-2ec5-43b2-b587-81e585865d08\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vd8tj"
Oct 30 19:16:08.803022 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.802753    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-registry-tls\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:08.803022 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.802780    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/247ff425-bed3-4297-b5c0-cc32b21bc179-service-ca-bundle\") pod \"router-default-75866dc98c-4mnhf\" (UID: \"247ff425-bed3-4297-b5c0-cc32b21bc179\") " pod="openshift-ingress/router-default-75866dc98c-4mnhf"
Oct 30 19:16:08.803022 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.802805    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28sgd\" (UniqueName: \"kubernetes.io/projected/247ff425-bed3-4297-b5c0-cc32b21bc179-kube-api-access-28sgd\") pod \"router-default-75866dc98c-4mnhf\" (UID: \"247ff425-bed3-4297-b5c0-cc32b21bc179\") " pod="openshift-ingress/router-default-75866dc98c-4mnhf"
Oct 30 19:16:08.803022 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.802841    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/76e21916-0744-4499-a0cb-64a7a44a8285-trusted-ca\") pod \"console-operator-7cd56b8dbc-cmw6w\" (UID: \"76e21916-0744-4499-a0cb-64a7a44a8285\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cmw6w"
Oct 30 19:16:08.803654 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.803606    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/247ff425-bed3-4297-b5c0-cc32b21bc179-default-certificate\") pod \"router-default-75866dc98c-4mnhf\" (UID: \"247ff425-bed3-4297-b5c0-cc32b21bc179\") " pod="openshift-ingress/router-default-75866dc98c-4mnhf"
Oct 30 19:16:08.803726 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:08.803710    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:16:08.803806 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:08.803782    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0269ffa6-ce06-43f3-be2a-58f6aa3e6964-metrics-tls podName:0269ffa6-ce06-43f3-be2a-58f6aa3e6964 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:09.303767113 +0000 UTC m=+31.111948107 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0269ffa6-ce06-43f3-be2a-58f6aa3e6964-metrics-tls") pod "dns-default-gkn6q" (UID: "0269ffa6-ce06-43f3-be2a-58f6aa3e6964") : secret "dns-default-metrics-tls" not found
Oct 30 19:16:08.803865 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.803806    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/76e21916-0744-4499-a0cb-64a7a44a8285-serving-cert\") pod \"console-operator-7cd56b8dbc-cmw6w\" (UID: \"76e21916-0744-4499-a0cb-64a7a44a8285\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cmw6w"
Oct 30 19:16:08.803865 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.803815    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/76e21916-0744-4499-a0cb-64a7a44a8285-trusted-ca\") pod \"console-operator-7cd56b8dbc-cmw6w\" (UID: \"76e21916-0744-4499-a0cb-64a7a44a8285\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cmw6w"
Oct 30 19:16:08.803963 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:08.803910    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:16:08.803963 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:08.803925    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78dc458d47-tqn8s: secret "image-registry-tls" not found
Oct 30 19:16:08.804061 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:08.803967    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-registry-tls podName:a64211d0-1c48-41d2-b254-4ae84d5097d7 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:09.30395258 +0000 UTC m=+31.112133575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-registry-tls") pod "image-registry-78dc458d47-tqn8s" (UID: "a64211d0-1c48-41d2-b254-4ae84d5097d7") : secret "image-registry-tls" not found
Oct 30 19:16:08.804061 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:08.804051    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/247ff425-bed3-4297-b5c0-cc32b21bc179-service-ca-bundle podName:247ff425-bed3-4297-b5c0-cc32b21bc179 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:09.30403892 +0000 UTC m=+31.112219935 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/247ff425-bed3-4297-b5c0-cc32b21bc179-service-ca-bundle") pod "router-default-75866dc98c-4mnhf" (UID: "247ff425-bed3-4297-b5c0-cc32b21bc179") : configmap references non-existent config key: service-ca.crt
Oct 30 19:16:08.804210 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.804186    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/06397fd4-3377-4eaa-8879-ebbb5943fbda-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-2sdkw\" (UID: \"06397fd4-3377-4eaa-8879-ebbb5943fbda\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2sdkw"
Oct 30 19:16:08.804315 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.804296    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/765d8d9d-b89f-419f-b237-3f8034db1bed-cert\") pod \"ingress-canary-9jkcb\" (UID: \"765d8d9d-b89f-419f-b237-3f8034db1bed\") " pod="openshift-ingress-canary/ingress-canary-9jkcb"
Oct 30 19:16:08.804374 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.804342    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/68201777-2ec5-43b2-b587-81e585865d08-config\") pod \"service-ca-operator-544d749c8c-vd8tj\" (UID: \"68201777-2ec5-43b2-b587-81e585865d08\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vd8tj"
Oct 30 19:16:08.804446 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:08.804430    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:16:08.804502 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:08.804477    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06397fd4-3377-4eaa-8879-ebbb5943fbda-cluster-monitoring-operator-tls podName:06397fd4-3377-4eaa-8879-ebbb5943fbda nodeName:}" failed. No retries permitted until 2026-10-30 19:16:09.304464241 +0000 UTC m=+31.112645238 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/06397fd4-3377-4eaa-8879-ebbb5943fbda-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2sdkw" (UID: "06397fd4-3377-4eaa-8879-ebbb5943fbda") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:16:08.805057 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.805029    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a64211d0-1c48-41d2-b254-4ae84d5097d7-trusted-ca\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:08.805057 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.805031    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a85cca66-a438-4117-9f50-977c7f528b7c-nginx-conf\") pod \"networking-console-plugin-86c478bb88-lpg7w\" (UID: \"a85cca66-a438-4117-9f50-977c7f528b7c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lpg7w"
Oct 30 19:16:08.805442 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.805420    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a64211d0-1c48-41d2-b254-4ae84d5097d7-registry-certificates\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:08.806357 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.806329    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a64211d0-1c48-41d2-b254-4ae84d5097d7-image-registry-private-configuration\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:08.806450 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.806355    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9f302411-01be-438c-a30c-70c944afd630-trusted-ca\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:08.806721 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.806685    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/353ebfb4-7e11-41de-b992-277443f56960-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-62bhw\" (UID: \"353ebfb4-7e11-41de-b992-277443f56960\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-62bhw"
Oct 30 19:16:08.807287 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.807261    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9f302411-01be-438c-a30c-70c944afd630-installation-pull-secrets\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:08.807473 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.807456    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68201777-2ec5-43b2-b587-81e585865d08-serving-cert\") pod \"service-ca-operator-544d749c8c-vd8tj\" (UID: \"68201777-2ec5-43b2-b587-81e585865d08\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vd8tj"
Oct 30 19:16:08.807616 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.807598    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e5002691-8257-4dab-a88b-3c1b60e8e6f8-serving-cert\") pod \"insights-operator-7c4986d449-fz9hf\" (UID: \"e5002691-8257-4dab-a88b-3c1b60e8e6f8\") " pod="openshift-insights/insights-operator-7c4986d449-fz9hf"
Oct 30 19:16:08.807771 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.807727    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9f302411-01be-438c-a30c-70c944afd630-image-registry-private-configuration\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:08.808012 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.807997    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a64211d0-1c48-41d2-b254-4ae84d5097d7-installation-pull-secrets\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:08.809364 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.809339    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/247ff425-bed3-4297-b5c0-cc32b21bc179-stats-auth\") pod \"router-default-75866dc98c-4mnhf\" (UID: \"247ff425-bed3-4297-b5c0-cc32b21bc179\") " pod="openshift-ingress/router-default-75866dc98c-4mnhf"
Oct 30 19:16:08.824908 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.824888    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh979\" (UniqueName: \"kubernetes.io/projected/06397fd4-3377-4eaa-8879-ebbb5943fbda-kube-api-access-zh979\") pod \"cluster-monitoring-operator-884576c57-2sdkw\" (UID: \"06397fd4-3377-4eaa-8879-ebbb5943fbda\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2sdkw"
Oct 30 19:16:08.825064 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.825047    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x795\" (UniqueName: \"kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-kube-api-access-6x795\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:08.827702 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.827654    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-bound-sa-token\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:08.829177 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.828900    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rz2d\" (UniqueName: \"kubernetes.io/projected/e5002691-8257-4dab-a88b-3c1b60e8e6f8-kube-api-access-9rz2d\") pod \"insights-operator-7c4986d449-fz9hf\" (UID: \"e5002691-8257-4dab-a88b-3c1b60e8e6f8\") " pod="openshift-insights/insights-operator-7c4986d449-fz9hf"
Oct 30 19:16:08.829177 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.829065    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-bound-sa-token\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:08.829177 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.829133    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tnx8\" (UniqueName: \"kubernetes.io/projected/c243f1d8-1c43-4eb8-93ec-cb28d72cc8a1-kube-api-access-4tnx8\") pod \"volume-data-source-validator-58c9f5f46d-mg98j\" (UID: \"c243f1d8-1c43-4eb8-93ec-cb28d72cc8a1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-mg98j"
Oct 30 19:16:08.829410 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.829383    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pfsl\" (UniqueName: \"kubernetes.io/projected/353ebfb4-7e11-41de-b992-277443f56960-kube-api-access-2pfsl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-62bhw\" (UID: \"353ebfb4-7e11-41de-b992-277443f56960\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-62bhw"
Oct 30 19:16:08.829504 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.829484    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfglw\" (UniqueName: \"kubernetes.io/projected/2d52a94d-72ac-4e5c-9472-104399a21a3b-kube-api-access-zfglw\") pod \"network-check-source-6f78d59f99-bvcb4\" (UID: \"2d52a94d-72ac-4e5c-9472-104399a21a3b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bvcb4"
Oct 30 19:16:08.831465 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.830566    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s56kf\" (UniqueName: \"kubernetes.io/projected/68201777-2ec5-43b2-b587-81e585865d08-kube-api-access-s56kf\") pod \"service-ca-operator-544d749c8c-vd8tj\" (UID: \"68201777-2ec5-43b2-b587-81e585865d08\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vd8tj"
Oct 30 19:16:08.831465 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.831120    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6chk\" (UniqueName: \"kubernetes.io/projected/76e21916-0744-4499-a0cb-64a7a44a8285-kube-api-access-v6chk\") pod \"console-operator-7cd56b8dbc-cmw6w\" (UID: \"76e21916-0744-4499-a0cb-64a7a44a8285\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cmw6w"
Oct 30 19:16:08.831465 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.831430    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njpgh\" (UniqueName: \"kubernetes.io/projected/765d8d9d-b89f-419f-b237-3f8034db1bed-kube-api-access-njpgh\") pod \"ingress-canary-9jkcb\" (UID: \"765d8d9d-b89f-419f-b237-3f8034db1bed\") " pod="openshift-ingress-canary/ingress-canary-9jkcb"
Oct 30 19:16:08.831465 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.831454    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpb5t\" (UniqueName: \"kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-kube-api-access-hpb5t\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:08.831860 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.831844    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6bsd\" (UniqueName: \"kubernetes.io/projected/0269ffa6-ce06-43f3-be2a-58f6aa3e6964-kube-api-access-b6bsd\") pod \"dns-default-gkn6q\" (UID: \"0269ffa6-ce06-43f3-be2a-58f6aa3e6964\") " pod="openshift-dns/dns-default-gkn6q"
Oct 30 19:16:08.836554 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.836532    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28sgd\" (UniqueName: \"kubernetes.io/projected/247ff425-bed3-4297-b5c0-cc32b21bc179-kube-api-access-28sgd\") pod \"router-default-75866dc98c-4mnhf\" (UID: \"247ff425-bed3-4297-b5c0-cc32b21bc179\") " pod="openshift-ingress/router-default-75866dc98c-4mnhf"
Oct 30 19:16:08.836764 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.836725    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcgjp\" (UniqueName: \"kubernetes.io/projected/cbc4f7c7-bebd-4200-833e-1afbf3c2c81a-kube-api-access-mcgjp\") pod \"cluster-samples-operator-6ddb6c8b94-p5nzf\" (UID: \"cbc4f7c7-bebd-4200-833e-1afbf3c2c81a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p5nzf"
Oct 30 19:16:08.836956 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.836940    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsjcc\" (UniqueName: \"kubernetes.io/projected/c84a5c22-7a46-4de1-aef7-32cb9a5453bb-kube-api-access-zsjcc\") pod \"kas-connection-checker-685895c7c9-zqvrw\" (UID: \"c84a5c22-7a46-4de1-aef7-32cb9a5453bb\") " pod="kube-system/kas-connection-checker-685895c7c9-zqvrw"
Oct 30 19:16:08.844896 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.844882    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-65kc2"
Oct 30 19:16:08.859856 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.859828    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6b8sd"
Oct 30 19:16:08.874049 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.874024    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-zqvrw"
Oct 30 19:16:08.899714 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.899198    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-cmw6w"
Oct 30 19:16:08.913196 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.911974    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-fz9hf"
Oct 30 19:16:08.928669 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.928642    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-62bhw"
Oct 30 19:16:08.938087 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.938058    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-mg98j"
Oct 30 19:16:08.955624 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.955172    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vd8tj"
Oct 30 19:16:08.968585 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.968488    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bvcb4"
Oct 30 19:16:08.983268 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:08.982891    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9jkcb"
Oct 30 19:16:09.135258 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:09.135229    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-65kc2"]
Oct 30 19:16:09.138466 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:16:09.138440    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9201f6f2_73b9_45cd_9950_3a07914d8ea9.slice/crio-14b01e2ba44a03c9687bf7ae9ad7d318310a5472f8b4a12c835a6894a525714a WatchSource:0}: Error finding container 14b01e2ba44a03c9687bf7ae9ad7d318310a5472f8b4a12c835a6894a525714a: Status 404 returned error can't find the container with id 14b01e2ba44a03c9687bf7ae9ad7d318310a5472f8b4a12c835a6894a525714a
Oct 30 19:16:09.145004 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:09.144528    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6b8sd"]
Oct 30 19:16:09.149931 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:16:09.149896    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00b97df0_857e_4ad9_b33a_4a59777b6f62.slice/crio-751934796f1ae1679278572b9ef2bb6bb010c50c66c514be4b59fa4e90ca381c WatchSource:0}: Error finding container 751934796f1ae1679278572b9ef2bb6bb010c50c66c514be4b59fa4e90ca381c: Status 404 returned error can't find the container with id 751934796f1ae1679278572b9ef2bb6bb010c50c66c514be4b59fa4e90ca381c
Oct 30 19:16:09.175201 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:09.175165    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-zqvrw"]
Oct 30 19:16:09.179315 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:16:09.179293    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc84a5c22_7a46_4de1_aef7_32cb9a5453bb.slice/crio-fb1b480febdfaa3569526b9c3c98e490125407a1721d4f174e0eb8d77d707b74 WatchSource:0}: Error finding container fb1b480febdfaa3569526b9c3c98e490125407a1721d4f174e0eb8d77d707b74: Status 404 returned error can't find the container with id fb1b480febdfaa3569526b9c3c98e490125407a1721d4f174e0eb8d77d707b74
Oct 30 19:16:09.228974 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:09.228943    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-fz9hf"]
Oct 30 19:16:09.232987 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:09.232936    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-cmw6w"]
Oct 30 19:16:09.236593 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:16:09.236504    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5002691_8257_4dab_a88b_3c1b60e8e6f8.slice/crio-ad4d9072a8cb0ef7d9197f2a1afef0c121f90b30eefd4f2ec780bfec7a063d4c WatchSource:0}: Error finding container ad4d9072a8cb0ef7d9197f2a1afef0c121f90b30eefd4f2ec780bfec7a063d4c: Status 404 returned error can't find the container with id ad4d9072a8cb0ef7d9197f2a1afef0c121f90b30eefd4f2ec780bfec7a063d4c
Oct 30 19:16:09.236945 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:16:09.236919    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76e21916_0744_4499_a0cb_64a7a44a8285.slice/crio-ca59d13272efc207f8bd73365bf08e0afe178dc4ec377b5ff544d3d9468374a7 WatchSource:0}: Error finding container ca59d13272efc207f8bd73365bf08e0afe178dc4ec377b5ff544d3d9468374a7: Status 404 returned error can't find the container with id ca59d13272efc207f8bd73365bf08e0afe178dc4ec377b5ff544d3d9468374a7
Oct 30 19:16:09.276482 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:09.276410    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-mg98j"]
Oct 30 19:16:09.281268 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:09.281115    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-vd8tj"]
Oct 30 19:16:09.282358 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:09.282332    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-62bhw"]
Oct 30 19:16:09.284172 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:16:09.283977    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68201777_2ec5_43b2_b587_81e585865d08.slice/crio-b4cf0b0eda03520a427f16b5b09773069d86edc6466c727314bbc46f8bdfb62f WatchSource:0}: Error finding container b4cf0b0eda03520a427f16b5b09773069d86edc6466c727314bbc46f8bdfb62f: Status 404 returned error can't find the container with id b4cf0b0eda03520a427f16b5b09773069d86edc6466c727314bbc46f8bdfb62f
Oct 30 19:16:09.290833 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:16:09.290807    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod353ebfb4_7e11_41de_b992_277443f56960.slice/crio-c7612191c1ecc48d5216a96ecbbc00bea70bdd93f79957dcfef867e1f359dbf9 WatchSource:0}: Error finding container c7612191c1ecc48d5216a96ecbbc00bea70bdd93f79957dcfef867e1f359dbf9: Status 404 returned error can't find the container with id c7612191c1ecc48d5216a96ecbbc00bea70bdd93f79957dcfef867e1f359dbf9
Oct 30 19:16:09.328003 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:09.327877    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-bvcb4"]
Oct 30 19:16:09.330684 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:09.330642    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/06397fd4-3377-4eaa-8879-ebbb5943fbda-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2sdkw\" (UID: \"06397fd4-3377-4eaa-8879-ebbb5943fbda\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2sdkw"
Oct 30 19:16:09.330834 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:09.330702    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-registry-tls\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:09.330834 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:09.330728    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/247ff425-bed3-4297-b5c0-cc32b21bc179-service-ca-bundle\") pod \"router-default-75866dc98c-4mnhf\" (UID: \"247ff425-bed3-4297-b5c0-cc32b21bc179\") " pod="openshift-ingress/router-default-75866dc98c-4mnhf"
Oct 30 19:16:09.330946 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:09.330830    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cbc4f7c7-bebd-4200-833e-1afbf3c2c81a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-p5nzf\" (UID: \"cbc4f7c7-bebd-4200-833e-1afbf3c2c81a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p5nzf"
Oct 30 19:16:09.330946 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:09.330873    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-registry-tls\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:09.330946 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:09.330902    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/247ff425-bed3-4297-b5c0-cc32b21bc179-metrics-certs\") pod \"router-default-75866dc98c-4mnhf\" (UID: \"247ff425-bed3-4297-b5c0-cc32b21bc179\") " pod="openshift-ingress/router-default-75866dc98c-4mnhf"
Oct 30 19:16:09.331085 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:09.330948    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a85cca66-a438-4117-9f50-977c7f528b7c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lpg7w\" (UID: \"a85cca66-a438-4117-9f50-977c7f528b7c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lpg7w"
Oct 30 19:16:09.331085 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:09.330987    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0269ffa6-ce06-43f3-be2a-58f6aa3e6964-metrics-tls\") pod \"dns-default-gkn6q\" (UID: \"0269ffa6-ce06-43f3-be2a-58f6aa3e6964\") " pod="openshift-dns/dns-default-gkn6q"
Oct 30 19:16:09.331181 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:09.331106    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:16:09.331181 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:09.331159    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0269ffa6-ce06-43f3-be2a-58f6aa3e6964-metrics-tls podName:0269ffa6-ce06-43f3-be2a-58f6aa3e6964 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:10.331142387 +0000 UTC m=+32.139323398 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0269ffa6-ce06-43f3-be2a-58f6aa3e6964-metrics-tls") pod "dns-default-gkn6q" (UID: "0269ffa6-ce06-43f3-be2a-58f6aa3e6964") : secret "dns-default-metrics-tls" not found
Oct 30 19:16:09.331316 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:09.331224    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:16:09.331316 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:09.331258    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06397fd4-3377-4eaa-8879-ebbb5943fbda-cluster-monitoring-operator-tls podName:06397fd4-3377-4eaa-8879-ebbb5943fbda nodeName:}" failed. No retries permitted until 2026-10-30 19:16:10.331246933 +0000 UTC m=+32.139427942 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/06397fd4-3377-4eaa-8879-ebbb5943fbda-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2sdkw" (UID: "06397fd4-3377-4eaa-8879-ebbb5943fbda") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:16:09.331316 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:09.331315    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:16:09.331464 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:09.331326    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78dc458d47-tqn8s: secret "image-registry-tls" not found
Oct 30 19:16:09.331464 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:09.331356    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-registry-tls podName:a64211d0-1c48-41d2-b254-4ae84d5097d7 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:10.331345847 +0000 UTC m=+32.139526855 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-registry-tls") pod "image-registry-78dc458d47-tqn8s" (UID: "a64211d0-1c48-41d2-b254-4ae84d5097d7") : secret "image-registry-tls" not found
Oct 30 19:16:09.331464 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:09.331417    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/247ff425-bed3-4297-b5c0-cc32b21bc179-service-ca-bundle podName:247ff425-bed3-4297-b5c0-cc32b21bc179 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:10.331407698 +0000 UTC m=+32.139588707 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/247ff425-bed3-4297-b5c0-cc32b21bc179-service-ca-bundle") pod "router-default-75866dc98c-4mnhf" (UID: "247ff425-bed3-4297-b5c0-cc32b21bc179") : configmap references non-existent config key: service-ca.crt
Oct 30 19:16:09.331644 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:09.331468    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 19:16:09.331644 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:09.331497    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbc4f7c7-bebd-4200-833e-1afbf3c2c81a-samples-operator-tls podName:cbc4f7c7-bebd-4200-833e-1afbf3c2c81a nodeName:}" failed. No retries permitted until 2026-10-30 19:16:10.331488128 +0000 UTC m=+32.139669136 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cbc4f7c7-bebd-4200-833e-1afbf3c2c81a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-p5nzf" (UID: "cbc4f7c7-bebd-4200-833e-1afbf3c2c81a") : secret "samples-operator-tls" not found
Oct 30 19:16:09.331644 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:09.331553    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:16:09.331644 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:09.331563    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-748f8689f4-6lwjn: secret "image-registry-tls" not found
Oct 30 19:16:09.331644 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:09.331590    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-registry-tls podName:9f302411-01be-438c-a30c-70c944afd630 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:10.33158085 +0000 UTC m=+32.139761857 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-registry-tls") pod "image-registry-748f8689f4-6lwjn" (UID: "9f302411-01be-438c-a30c-70c944afd630") : secret "image-registry-tls" not found
Oct 30 19:16:09.331644 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:09.331644    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:16:09.331937 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:09.331671    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/247ff425-bed3-4297-b5c0-cc32b21bc179-metrics-certs podName:247ff425-bed3-4297-b5c0-cc32b21bc179 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:10.331662308 +0000 UTC m=+32.139843316 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/247ff425-bed3-4297-b5c0-cc32b21bc179-metrics-certs") pod "router-default-75866dc98c-4mnhf" (UID: "247ff425-bed3-4297-b5c0-cc32b21bc179") : secret "router-metrics-certs-default" not found
Oct 30 19:16:09.331937 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:09.331723    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:16:09.331937 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:09.331784    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a85cca66-a438-4117-9f50-977c7f528b7c-networking-console-plugin-cert podName:a85cca66-a438-4117-9f50-977c7f528b7c nodeName:}" failed. No retries permitted until 2026-10-30 19:16:10.331774549 +0000 UTC m=+32.139955557 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a85cca66-a438-4117-9f50-977c7f528b7c-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-lpg7w" (UID: "a85cca66-a438-4117-9f50-977c7f528b7c") : secret "networking-console-plugin-cert" not found
Oct 30 19:16:09.333073 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:09.332465    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9jkcb"]
Oct 30 19:16:09.335146 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:16:09.335125    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod765d8d9d_b89f_419f_b237_3f8034db1bed.slice/crio-fe12e0dde11c16433babd16be15e78a1bce1c00b6854cf7addaa3503d314dd5a WatchSource:0}: Error finding container fe12e0dde11c16433babd16be15e78a1bce1c00b6854cf7addaa3503d314dd5a: Status 404 returned error can't find the container with id fe12e0dde11c16433babd16be15e78a1bce1c00b6854cf7addaa3503d314dd5a
Oct 30 19:16:09.645125 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:09.645046    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9cws"
Oct 30 19:16:09.645125 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:09.645090    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wg2df"
Oct 30 19:16:09.645274 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:09.645045    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n9zx2"
Oct 30 19:16:09.649026 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:09.649006    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2k8z2\""
Oct 30 19:16:09.649026 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:09.649008    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 19:16:09.649378 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:09.649077    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-6bckm\""
Oct 30 19:16:09.649378 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:09.649138    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-2qhvn\""
Oct 30 19:16:09.649378 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:09.649250    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 19:16:09.907776 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:09.907659    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-62bhw" event={"ID":"353ebfb4-7e11-41de-b992-277443f56960","Type":"ContainerStarted","Data":"c7612191c1ecc48d5216a96ecbbc00bea70bdd93f79957dcfef867e1f359dbf9"}
Oct 30 19:16:09.910044 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:09.910007    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6b8sd" event={"ID":"00b97df0-857e-4ad9-b33a-4a59777b6f62","Type":"ContainerStarted","Data":"533ca6a200d83f91d2ca6e4831477488773290044912d64da4132f73838d749f"}
Oct 30 19:16:09.910165 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:09.910061    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6b8sd" event={"ID":"00b97df0-857e-4ad9-b33a-4a59777b6f62","Type":"ContainerStarted","Data":"751934796f1ae1679278572b9ef2bb6bb010c50c66c514be4b59fa4e90ca381c"}
Oct 30 19:16:09.911956 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:09.911928    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bvcb4" event={"ID":"2d52a94d-72ac-4e5c-9472-104399a21a3b","Type":"ContainerStarted","Data":"5b80a342a983dcc8d8c93e5a7498eb7deed13dd3ae040b10a76c10a2c4c52644"}
Oct 30 19:16:09.913911 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:09.913867    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-mg98j" event={"ID":"c243f1d8-1c43-4eb8-93ec-cb28d72cc8a1","Type":"ContainerStarted","Data":"c8d548b04e3d8e8c629730de6fc83598257db1ec337aabbb0514d1d0d007a668"}
Oct 30 19:16:09.916059 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:09.916000    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-fz9hf" event={"ID":"e5002691-8257-4dab-a88b-3c1b60e8e6f8","Type":"ContainerStarted","Data":"ad4d9072a8cb0ef7d9197f2a1afef0c121f90b30eefd4f2ec780bfec7a063d4c"}
Oct 30 19:16:09.919607 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:09.919517    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-zqvrw" event={"ID":"c84a5c22-7a46-4de1-aef7-32cb9a5453bb","Type":"ContainerStarted","Data":"5dbd7d7f1ce326cc80cd79f30cb1c7c6927e5a02e75ad7534a1af82b12e978b3"}
Oct 30 19:16:09.919607 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:09.919566    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-zqvrw" event={"ID":"c84a5c22-7a46-4de1-aef7-32cb9a5453bb","Type":"ContainerStarted","Data":"fb1b480febdfaa3569526b9c3c98e490125407a1721d4f174e0eb8d77d707b74"}
Oct 30 19:16:09.921402 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:09.921376    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9jkcb" event={"ID":"765d8d9d-b89f-419f-b237-3f8034db1bed","Type":"ContainerStarted","Data":"fe12e0dde11c16433babd16be15e78a1bce1c00b6854cf7addaa3503d314dd5a"}
Oct 30 19:16:09.922897 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:09.922826    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vd8tj" event={"ID":"68201777-2ec5-43b2-b587-81e585865d08","Type":"ContainerStarted","Data":"b4cf0b0eda03520a427f16b5b09773069d86edc6466c727314bbc46f8bdfb62f"}
Oct 30 19:16:09.924247 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:09.924224    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-cmw6w" event={"ID":"76e21916-0744-4499-a0cb-64a7a44a8285","Type":"ContainerStarted","Data":"ca59d13272efc207f8bd73365bf08e0afe178dc4ec377b5ff544d3d9468374a7"}
Oct 30 19:16:09.926263 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:09.926237    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-65kc2" event={"ID":"9201f6f2-73b9-45cd-9950-3a07914d8ea9","Type":"ContainerStarted","Data":"a59725f28e77015f849035a22f96efb0de28eb926fb5de1fb807ee86950d9cc9"}
Oct 30 19:16:09.926366 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:09.926282    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-65kc2" event={"ID":"9201f6f2-73b9-45cd-9950-3a07914d8ea9","Type":"ContainerStarted","Data":"14b01e2ba44a03c9687bf7ae9ad7d318310a5472f8b4a12c835a6894a525714a"}
Oct 30 19:16:09.929661 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:09.929614    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-6b8sd" podStartSLOduration=422.929602476 podStartE2EDuration="7m2.929602476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:09.928446564 +0000 UTC m=+31.736627580" watchObservedRunningTime="2026-10-30 19:16:09.929602476 +0000 UTC m=+31.737783491"
Oct 30 19:16:09.949770 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:09.948629    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-zqvrw" podStartSLOduration=422.94861178 podStartE2EDuration="7m2.94861178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:09.947942814 +0000 UTC m=+31.756123831" watchObservedRunningTime="2026-10-30 19:16:09.94861178 +0000 UTC m=+31.756792797"
Oct 30 19:16:09.971254 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:09.971198    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-65kc2" podStartSLOduration=422.971177876 podStartE2EDuration="7m2.971177876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:09.970507972 +0000 UTC m=+31.778688991" watchObservedRunningTime="2026-10-30 19:16:09.971177876 +0000 UTC m=+31.779358893"
Oct 30 19:16:10.340623 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:10.340581    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/06397fd4-3377-4eaa-8879-ebbb5943fbda-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2sdkw\" (UID: \"06397fd4-3377-4eaa-8879-ebbb5943fbda\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2sdkw"
Oct 30 19:16:10.340798 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:10.340642    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-registry-tls\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:10.340798 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:10.340671    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/247ff425-bed3-4297-b5c0-cc32b21bc179-service-ca-bundle\") pod \"router-default-75866dc98c-4mnhf\" (UID: \"247ff425-bed3-4297-b5c0-cc32b21bc179\") " pod="openshift-ingress/router-default-75866dc98c-4mnhf"
Oct 30 19:16:10.340798 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:10.340789    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cbc4f7c7-bebd-4200-833e-1afbf3c2c81a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-p5nzf\" (UID: \"cbc4f7c7-bebd-4200-833e-1afbf3c2c81a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p5nzf"
Oct 30 19:16:10.340968 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:10.340832    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-registry-tls\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:10.340968 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:10.340861    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/247ff425-bed3-4297-b5c0-cc32b21bc179-metrics-certs\") pod \"router-default-75866dc98c-4mnhf\" (UID: \"247ff425-bed3-4297-b5c0-cc32b21bc179\") " pod="openshift-ingress/router-default-75866dc98c-4mnhf"
Oct 30 19:16:10.340968 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:10.340907    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a85cca66-a438-4117-9f50-977c7f528b7c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lpg7w\" (UID: \"a85cca66-a438-4117-9f50-977c7f528b7c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lpg7w"
Oct 30 19:16:10.340968 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:10.340947    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0269ffa6-ce06-43f3-be2a-58f6aa3e6964-metrics-tls\") pod \"dns-default-gkn6q\" (UID: \"0269ffa6-ce06-43f3-be2a-58f6aa3e6964\") " pod="openshift-dns/dns-default-gkn6q"
Oct 30 19:16:10.341190 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:10.341176    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:16:10.341330 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:10.341241    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0269ffa6-ce06-43f3-be2a-58f6aa3e6964-metrics-tls podName:0269ffa6-ce06-43f3-be2a-58f6aa3e6964 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:12.341222968 +0000 UTC m=+34.149403964 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0269ffa6-ce06-43f3-be2a-58f6aa3e6964-metrics-tls") pod "dns-default-gkn6q" (UID: "0269ffa6-ce06-43f3-be2a-58f6aa3e6964") : secret "dns-default-metrics-tls" not found
Oct 30 19:16:10.341661 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:10.341642    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:16:10.341723 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:10.341698    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06397fd4-3377-4eaa-8879-ebbb5943fbda-cluster-monitoring-operator-tls podName:06397fd4-3377-4eaa-8879-ebbb5943fbda nodeName:}" failed. No retries permitted until 2026-10-30 19:16:12.341684223 +0000 UTC m=+34.149865223 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/06397fd4-3377-4eaa-8879-ebbb5943fbda-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2sdkw" (UID: "06397fd4-3377-4eaa-8879-ebbb5943fbda") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:16:10.341808 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:10.341781    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:16:10.341808 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:10.341793    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78dc458d47-tqn8s: secret "image-registry-tls" not found
Oct 30 19:16:10.341904 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:10.341829    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-registry-tls podName:a64211d0-1c48-41d2-b254-4ae84d5097d7 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:12.341817557 +0000 UTC m=+34.149998554 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-registry-tls") pod "image-registry-78dc458d47-tqn8s" (UID: "a64211d0-1c48-41d2-b254-4ae84d5097d7") : secret "image-registry-tls" not found
Oct 30 19:16:10.341990 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:10.341893    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/247ff425-bed3-4297-b5c0-cc32b21bc179-service-ca-bundle podName:247ff425-bed3-4297-b5c0-cc32b21bc179 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:12.341882236 +0000 UTC m=+34.150063234 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/247ff425-bed3-4297-b5c0-cc32b21bc179-service-ca-bundle") pod "router-default-75866dc98c-4mnhf" (UID: "247ff425-bed3-4297-b5c0-cc32b21bc179") : configmap references non-existent config key: service-ca.crt
Oct 30 19:16:10.341990 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:10.341962    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 19:16:10.342092 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:10.341994    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbc4f7c7-bebd-4200-833e-1afbf3c2c81a-samples-operator-tls podName:cbc4f7c7-bebd-4200-833e-1afbf3c2c81a nodeName:}" failed. No retries permitted until 2026-10-30 19:16:12.341982944 +0000 UTC m=+34.150163942 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cbc4f7c7-bebd-4200-833e-1afbf3c2c81a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-p5nzf" (UID: "cbc4f7c7-bebd-4200-833e-1afbf3c2c81a") : secret "samples-operator-tls" not found
Oct 30 19:16:10.342144 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:10.342093    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:16:10.342144 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:10.342104    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-748f8689f4-6lwjn: secret "image-registry-tls" not found
Oct 30 19:16:10.342144 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:10.342144    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-registry-tls podName:9f302411-01be-438c-a30c-70c944afd630 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:12.342134294 +0000 UTC m=+34.150315301 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-registry-tls") pod "image-registry-748f8689f4-6lwjn" (UID: "9f302411-01be-438c-a30c-70c944afd630") : secret "image-registry-tls" not found
Oct 30 19:16:10.342353 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:10.342338    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:16:10.342404 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:10.342389    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a85cca66-a438-4117-9f50-977c7f528b7c-networking-console-plugin-cert podName:a85cca66-a438-4117-9f50-977c7f528b7c nodeName:}" failed. No retries permitted until 2026-10-30 19:16:12.342376035 +0000 UTC m=+34.150557033 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a85cca66-a438-4117-9f50-977c7f528b7c-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-lpg7w" (UID: "a85cca66-a438-4117-9f50-977c7f528b7c") : secret "networking-console-plugin-cert" not found
Oct 30 19:16:10.342485 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:10.342204    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:16:10.342569 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:10.342528    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/247ff425-bed3-4297-b5c0-cc32b21bc179-metrics-certs podName:247ff425-bed3-4297-b5c0-cc32b21bc179 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:12.342516255 +0000 UTC m=+34.150697254 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/247ff425-bed3-4297-b5c0-cc32b21bc179-metrics-certs") pod "router-default-75866dc98c-4mnhf" (UID: "247ff425-bed3-4297-b5c0-cc32b21bc179") : secret "router-metrics-certs-default" not found
Oct 30 19:16:11.946192 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:11.945924    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-mg98j" event={"ID":"c243f1d8-1c43-4eb8-93ec-cb28d72cc8a1","Type":"ContainerStarted","Data":"f92206c3962c39f6d29a7a80e94cf02bd526cf59785b6487ad39a48b339302df"}
Oct 30 19:16:11.963030 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:11.962092    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-mg98j" podStartSLOduration=391.612529055 podStartE2EDuration="6m33.962073472s" totalImagesPullingTime="2.349544417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:11.961168647 +0000 UTC m=+33.769349664" watchObservedRunningTime="2026-10-30 19:16:11.962073472 +0000 UTC m=+33.770254489"
Oct 30 19:16:12.368090 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:12.368047    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cbc4f7c7-bebd-4200-833e-1afbf3c2c81a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-p5nzf\" (UID: \"cbc4f7c7-bebd-4200-833e-1afbf3c2c81a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p5nzf"
Oct 30 19:16:12.368275 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:12.368118    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-registry-tls\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:12.368275 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:12.368151    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/247ff425-bed3-4297-b5c0-cc32b21bc179-metrics-certs\") pod \"router-default-75866dc98c-4mnhf\" (UID: \"247ff425-bed3-4297-b5c0-cc32b21bc179\") " pod="openshift-ingress/router-default-75866dc98c-4mnhf"
Oct 30 19:16:12.368275 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:12.368199    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a85cca66-a438-4117-9f50-977c7f528b7c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lpg7w\" (UID: \"a85cca66-a438-4117-9f50-977c7f528b7c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lpg7w"
Oct 30 19:16:12.368275 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:12.368236    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0269ffa6-ce06-43f3-be2a-58f6aa3e6964-metrics-tls\") pod \"dns-default-gkn6q\" (UID: \"0269ffa6-ce06-43f3-be2a-58f6aa3e6964\") " pod="openshift-dns/dns-default-gkn6q"
Oct 30 19:16:12.368475 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:12.368287    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/06397fd4-3377-4eaa-8879-ebbb5943fbda-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2sdkw\" (UID: \"06397fd4-3377-4eaa-8879-ebbb5943fbda\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2sdkw"
Oct 30 19:16:12.368475 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:12.368316    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-registry-tls\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:12.368475 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:12.368341    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/247ff425-bed3-4297-b5c0-cc32b21bc179-service-ca-bundle\") pod \"router-default-75866dc98c-4mnhf\" (UID: \"247ff425-bed3-4297-b5c0-cc32b21bc179\") " pod="openshift-ingress/router-default-75866dc98c-4mnhf"
Oct 30 19:16:12.368604 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:12.368547    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/247ff425-bed3-4297-b5c0-cc32b21bc179-service-ca-bundle podName:247ff425-bed3-4297-b5c0-cc32b21bc179 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:16.368527594 +0000 UTC m=+38.176708610 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/247ff425-bed3-4297-b5c0-cc32b21bc179-service-ca-bundle") pod "router-default-75866dc98c-4mnhf" (UID: "247ff425-bed3-4297-b5c0-cc32b21bc179") : configmap references non-existent config key: service-ca.crt
Oct 30 19:16:12.369008 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:12.368982    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 19:16:12.369137 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:12.369042    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbc4f7c7-bebd-4200-833e-1afbf3c2c81a-samples-operator-tls podName:cbc4f7c7-bebd-4200-833e-1afbf3c2c81a nodeName:}" failed. No retries permitted until 2026-10-30 19:16:16.369026983 +0000 UTC m=+38.177207992 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cbc4f7c7-bebd-4200-833e-1afbf3c2c81a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-p5nzf" (UID: "cbc4f7c7-bebd-4200-833e-1afbf3c2c81a") : secret "samples-operator-tls" not found
Oct 30 19:16:12.369137 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:12.369103    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:16:12.369137 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:12.369114    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-748f8689f4-6lwjn: secret "image-registry-tls" not found
Oct 30 19:16:12.369306 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:12.369148    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-registry-tls podName:9f302411-01be-438c-a30c-70c944afd630 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:16.369136472 +0000 UTC m=+38.177317481 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-registry-tls") pod "image-registry-748f8689f4-6lwjn" (UID: "9f302411-01be-438c-a30c-70c944afd630") : secret "image-registry-tls" not found
Oct 30 19:16:12.369306 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:12.369203    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:16:12.369306 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:12.369236    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/247ff425-bed3-4297-b5c0-cc32b21bc179-metrics-certs podName:247ff425-bed3-4297-b5c0-cc32b21bc179 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:16.369225578 +0000 UTC m=+38.177406585 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/247ff425-bed3-4297-b5c0-cc32b21bc179-metrics-certs") pod "router-default-75866dc98c-4mnhf" (UID: "247ff425-bed3-4297-b5c0-cc32b21bc179") : secret "router-metrics-certs-default" not found
Oct 30 19:16:12.369306 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:12.369291    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:16:12.369525 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:12.369331    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a85cca66-a438-4117-9f50-977c7f528b7c-networking-console-plugin-cert podName:a85cca66-a438-4117-9f50-977c7f528b7c nodeName:}" failed. No retries permitted until 2026-10-30 19:16:16.369311624 +0000 UTC m=+38.177492632 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a85cca66-a438-4117-9f50-977c7f528b7c-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-lpg7w" (UID: "a85cca66-a438-4117-9f50-977c7f528b7c") : secret "networking-console-plugin-cert" not found
Oct 30 19:16:12.369525 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:12.369388    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:16:12.369525 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:12.369417    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0269ffa6-ce06-43f3-be2a-58f6aa3e6964-metrics-tls podName:0269ffa6-ce06-43f3-be2a-58f6aa3e6964 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:16.369407671 +0000 UTC m=+38.177588680 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0269ffa6-ce06-43f3-be2a-58f6aa3e6964-metrics-tls") pod "dns-default-gkn6q" (UID: "0269ffa6-ce06-43f3-be2a-58f6aa3e6964") : secret "dns-default-metrics-tls" not found
Oct 30 19:16:12.369525 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:12.369482    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:16:12.369525 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:12.369515    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06397fd4-3377-4eaa-8879-ebbb5943fbda-cluster-monitoring-operator-tls podName:06397fd4-3377-4eaa-8879-ebbb5943fbda nodeName:}" failed. No retries permitted until 2026-10-30 19:16:16.369504526 +0000 UTC m=+38.177685534 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/06397fd4-3377-4eaa-8879-ebbb5943fbda-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2sdkw" (UID: "06397fd4-3377-4eaa-8879-ebbb5943fbda") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:16:12.369837 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:12.369574    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:16:12.369837 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:12.369585    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78dc458d47-tqn8s: secret "image-registry-tls" not found
Oct 30 19:16:12.369837 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:12.369614    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-registry-tls podName:a64211d0-1c48-41d2-b254-4ae84d5097d7 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:16.369603893 +0000 UTC m=+38.177784901 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-registry-tls") pod "image-registry-78dc458d47-tqn8s" (UID: "a64211d0-1c48-41d2-b254-4ae84d5097d7") : secret "image-registry-tls" not found
Oct 30 19:16:12.469198 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:12.469134    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ecb643b-1dce-4523-89df-93b22ad8ce6f-original-pull-secret\") pod \"global-pull-secret-syncer-wg2df\" (UID: \"6ecb643b-1dce-4523-89df-93b22ad8ce6f\") " pod="kube-system/global-pull-secret-syncer-wg2df"
Oct 30 19:16:12.469374 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:12.469203    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8e297a2-a78a-4665-86d0-6d842bbcaa51-metrics-certs\") pod \"network-metrics-daemon-n9cws\" (UID: \"e8e297a2-a78a-4665-86d0-6d842bbcaa51\") " pod="openshift-multus/network-metrics-daemon-n9cws"
Oct 30 19:16:12.469513 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:12.469491    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 19:16:12.469582 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:12.469568    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8e297a2-a78a-4665-86d0-6d842bbcaa51-metrics-certs podName:e8e297a2-a78a-4665-86d0-6d842bbcaa51 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:44.469549562 +0000 UTC m=+66.277730561 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8e297a2-a78a-4665-86d0-6d842bbcaa51-metrics-certs") pod "network-metrics-daemon-n9cws" (UID: "e8e297a2-a78a-4665-86d0-6d842bbcaa51") : secret "metrics-daemon-secret" not found
Oct 30 19:16:12.473646 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:12.473618    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ecb643b-1dce-4523-89df-93b22ad8ce6f-original-pull-secret\") pod \"global-pull-secret-syncer-wg2df\" (UID: \"6ecb643b-1dce-4523-89df-93b22ad8ce6f\") " pod="kube-system/global-pull-secret-syncer-wg2df"
Oct 30 19:16:12.570004 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:12.569963    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gnlf\" (UniqueName: \"kubernetes.io/projected/47e9832e-bc96-445a-a93e-9d24f84db3b3-kube-api-access-6gnlf\") pod \"network-check-target-n9zx2\" (UID: \"47e9832e-bc96-445a-a93e-9d24f84db3b3\") " pod="openshift-network-diagnostics/network-check-target-n9zx2"
Oct 30 19:16:12.576005 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:12.575973    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gnlf\" (UniqueName: \"kubernetes.io/projected/47e9832e-bc96-445a-a93e-9d24f84db3b3-kube-api-access-6gnlf\") pod \"network-check-target-n9zx2\" (UID: \"47e9832e-bc96-445a-a93e-9d24f84db3b3\") " pod="openshift-network-diagnostics/network-check-target-n9zx2"
Oct 30 19:16:12.722584 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:12.722490    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wg2df"
Oct 30 19:16:12.739116 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:12.739084    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n9zx2"
Oct 30 19:16:13.069286 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:13.069128    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-n9zx2"]
Oct 30 19:16:13.072243 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:13.072222    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wg2df"]
Oct 30 19:16:13.220460 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:16:13.220423    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ecb643b_1dce_4523_89df_93b22ad8ce6f.slice/crio-c8faece2f3197dd85421545da5f23e9e2d01747c378c197afd296e70c026d12a WatchSource:0}: Error finding container c8faece2f3197dd85421545da5f23e9e2d01747c378c197afd296e70c026d12a: Status 404 returned error can't find the container with id c8faece2f3197dd85421545da5f23e9e2d01747c378c197afd296e70c026d12a
Oct 30 19:16:13.223374 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:16:13.222961    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47e9832e_bc96_445a_a93e_9d24f84db3b3.slice/crio-012f367ad67d33090ad1abbbf367716fe83ab03857802f0c325e08d1b002f679 WatchSource:0}: Error finding container 012f367ad67d33090ad1abbbf367716fe83ab03857802f0c325e08d1b002f679: Status 404 returned error can't find the container with id 012f367ad67d33090ad1abbbf367716fe83ab03857802f0c325e08d1b002f679
Oct 30 19:16:13.953861 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:13.953831    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cmw6w_76e21916-0744-4499-a0cb-64a7a44a8285/console-operator/0.log"
Oct 30 19:16:13.954025 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:13.953886    2579 generic.go:356] "Generic (PLEG): container finished" podID="76e21916-0744-4499-a0cb-64a7a44a8285" containerID="b7916ea13ba1e948549de79c688d0b1b459c379f737bfe829c904ea78adeef29" exitCode=255
Oct 30 19:16:13.954025 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:13.953982    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-cmw6w" event={"ID":"76e21916-0744-4499-a0cb-64a7a44a8285","Type":"ContainerDied","Data":"b7916ea13ba1e948549de79c688d0b1b459c379f737bfe829c904ea78adeef29"}
Oct 30 19:16:13.954210 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:13.954196    2579 scope.go:122] "RemoveContainer" containerID="b7916ea13ba1e948549de79c688d0b1b459c379f737bfe829c904ea78adeef29"
Oct 30 19:16:13.955728 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:13.955697    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-n9zx2" event={"ID":"47e9832e-bc96-445a-a93e-9d24f84db3b3","Type":"ContainerStarted","Data":"012f367ad67d33090ad1abbbf367716fe83ab03857802f0c325e08d1b002f679"}
Oct 30 19:16:13.957430 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:13.957408    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wg2df" event={"ID":"6ecb643b-1dce-4523-89df-93b22ad8ce6f","Type":"ContainerStarted","Data":"c8faece2f3197dd85421545da5f23e9e2d01747c378c197afd296e70c026d12a"}
Oct 30 19:16:14.965085 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:14.964397    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-fz9hf" event={"ID":"e5002691-8257-4dab-a88b-3c1b60e8e6f8","Type":"ContainerStarted","Data":"de80aadc9929788187e54b1f259f5300d637f50bdc7d72b6d616054e2fdd1196"}
Oct 30 19:16:14.967372 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:14.966953    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9jkcb" event={"ID":"765d8d9d-b89f-419f-b237-3f8034db1bed","Type":"ContainerStarted","Data":"80c2bc83b90b1ec87fcb657001dc71055a25d0cd9c73df035a8f1c1ca791a55a"}
Oct 30 19:16:14.969343 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:14.968986    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vd8tj" event={"ID":"68201777-2ec5-43b2-b587-81e585865d08","Type":"ContainerStarted","Data":"89510aef964a5db824e385074ed48b40781681da2cf2592625095bca48484db6"}
Oct 30 19:16:14.971792 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:14.971582    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cmw6w_76e21916-0744-4499-a0cb-64a7a44a8285/console-operator/1.log"
Oct 30 19:16:14.972799 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:14.972102    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cmw6w_76e21916-0744-4499-a0cb-64a7a44a8285/console-operator/0.log"
Oct 30 19:16:14.972799 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:14.972134    2579 generic.go:356] "Generic (PLEG): container finished" podID="76e21916-0744-4499-a0cb-64a7a44a8285" containerID="b3f38d74f7964cb2e0db345a5bf20e6f2b18ed1ffffa37aafe2a406f3f91103c" exitCode=255
Oct 30 19:16:14.972799 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:14.972225    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-cmw6w" event={"ID":"76e21916-0744-4499-a0cb-64a7a44a8285","Type":"ContainerDied","Data":"b3f38d74f7964cb2e0db345a5bf20e6f2b18ed1ffffa37aafe2a406f3f91103c"}
Oct 30 19:16:14.972799 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:14.972262    2579 scope.go:122] "RemoveContainer" containerID="b7916ea13ba1e948549de79c688d0b1b459c379f737bfe829c904ea78adeef29"
Oct 30 19:16:14.972799 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:14.972567    2579 scope.go:122] "RemoveContainer" containerID="b3f38d74f7964cb2e0db345a5bf20e6f2b18ed1ffffa37aafe2a406f3f91103c"
Oct 30 19:16:14.972799 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:14.972688    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-cmw6w_openshift-console-operator(76e21916-0744-4499-a0cb-64a7a44a8285)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-cmw6w" podUID="76e21916-0744-4499-a0cb-64a7a44a8285"
Oct 30 19:16:14.975202 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:14.974717    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-62bhw" event={"ID":"353ebfb4-7e11-41de-b992-277443f56960","Type":"ContainerStarted","Data":"4b81bc0bd6efd1290bc9ea02d3572f3b8ca96bb1198b289a762c5933e0449b4f"}
Oct 30 19:16:14.985534 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:14.984646    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-fz9hf" podStartSLOduration=391.890224766 podStartE2EDuration="6m36.984629253s" totalImagesPullingTime="5.094404487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:14.982824558 +0000 UTC m=+36.791005575" watchObservedRunningTime="2026-10-30 19:16:14.984629253 +0000 UTC m=+36.792810272"
Oct 30 19:16:15.036531 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.035079    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-62bhw" podStartSLOduration=391.985388759 podStartE2EDuration="6m37.035059255s" totalImagesPullingTime="5.049670496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:15.03259507 +0000 UTC m=+36.840776088" watchObservedRunningTime="2026-10-30 19:16:15.035059255 +0000 UTC m=+36.843240274"
Oct 30 19:16:15.077154 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.075822    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9jkcb" podStartSLOduration=1.6326382430000002 podStartE2EDuration="7.075803228s" totalImagesPullingTime="5.443164985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:15.074914325 +0000 UTC m=+36.883095342" watchObservedRunningTime="2026-10-30 19:16:15.075803228 +0000 UTC m=+36.883984245"
Oct 30 19:16:15.142412 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.139585    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vd8tj" podStartSLOduration=392.059971758 podStartE2EDuration="6m37.139564549s" totalImagesPullingTime="5.079592791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:15.093311642 +0000 UTC m=+36.901492659" watchObservedRunningTime="2026-10-30 19:16:15.139564549 +0000 UTC m=+36.947745569"
Oct 30 19:16:15.142412 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.140837    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-smgf6"]
Oct 30 19:16:15.158019 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.157356    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-smgf6"]
Oct 30 19:16:15.158019 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.157482    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-smgf6"
Oct 30 19:16:15.164181 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.162210    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-2lcwk-gjv8f"]
Oct 30 19:16:15.192728 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.192707    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 19:16:15.195538 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.195510    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-2lcwk-gjv8f"]
Oct 30 19:16:15.195654 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.195634    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-2lcwk-gjv8f"
Oct 30 19:16:15.219100 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.219075    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aeef13dc-e06e-49c5-b893-64c7f73a7e44-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-smgf6\" (UID: \"aeef13dc-e06e-49c5-b893-64c7f73a7e44\") " pod="openshift-insights/insights-runtime-extractor-smgf6"
Oct 30 19:16:15.219219 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.219116    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aeef13dc-e06e-49c5-b893-64c7f73a7e44-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-smgf6\" (UID: \"aeef13dc-e06e-49c5-b893-64c7f73a7e44\") " pod="openshift-insights/insights-runtime-extractor-smgf6"
Oct 30 19:16:15.219219 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.219157    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq4kp\" (UniqueName: \"kubernetes.io/projected/aeef13dc-e06e-49c5-b893-64c7f73a7e44-kube-api-access-tq4kp\") pod \"insights-runtime-extractor-smgf6\" (UID: \"aeef13dc-e06e-49c5-b893-64c7f73a7e44\") " pod="openshift-insights/insights-runtime-extractor-smgf6"
Oct 30 19:16:15.219326 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.219216    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aeef13dc-e06e-49c5-b893-64c7f73a7e44-data-volume\") pod \"insights-runtime-extractor-smgf6\" (UID: \"aeef13dc-e06e-49c5-b893-64c7f73a7e44\") " pod="openshift-insights/insights-runtime-extractor-smgf6"
Oct 30 19:16:15.219326 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.219262    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aeef13dc-e06e-49c5-b893-64c7f73a7e44-crio-socket\") pod \"insights-runtime-extractor-smgf6\" (UID: \"aeef13dc-e06e-49c5-b893-64c7f73a7e44\") " pod="openshift-insights/insights-runtime-extractor-smgf6"
Oct 30 19:16:15.222182 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.222096    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 19:16:15.222530 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.222371    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7b7kv\""
Oct 30 19:16:15.320041 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.320003    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aeef13dc-e06e-49c5-b893-64c7f73a7e44-crio-socket\") pod \"insights-runtime-extractor-smgf6\" (UID: \"aeef13dc-e06e-49c5-b893-64c7f73a7e44\") " pod="openshift-insights/insights-runtime-extractor-smgf6"
Oct 30 19:16:15.320192 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.320057    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xrrr\" (UniqueName: \"kubernetes.io/projected/5338b461-6b7c-474b-8458-d1b53775ee0f-kube-api-access-4xrrr\") pod \"periodic-gathering-2lcwk-gjv8f\" (UID: \"5338b461-6b7c-474b-8458-d1b53775ee0f\") " pod="openshift-insights/periodic-gathering-2lcwk-gjv8f"
Oct 30 19:16:15.320254 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.320233    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5338b461-6b7c-474b-8458-d1b53775ee0f-archives-path\") pod \"periodic-gathering-2lcwk-gjv8f\" (UID: \"5338b461-6b7c-474b-8458-d1b53775ee0f\") " pod="openshift-insights/periodic-gathering-2lcwk-gjv8f"
Oct 30 19:16:15.320313 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.320262    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aeef13dc-e06e-49c5-b893-64c7f73a7e44-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-smgf6\" (UID: \"aeef13dc-e06e-49c5-b893-64c7f73a7e44\") " pod="openshift-insights/insights-runtime-extractor-smgf6"
Oct 30 19:16:15.320313 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.320287    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aeef13dc-e06e-49c5-b893-64c7f73a7e44-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-smgf6\" (UID: \"aeef13dc-e06e-49c5-b893-64c7f73a7e44\") " pod="openshift-insights/insights-runtime-extractor-smgf6"
Oct 30 19:16:15.320399 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.320321    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq4kp\" (UniqueName: \"kubernetes.io/projected/aeef13dc-e06e-49c5-b893-64c7f73a7e44-kube-api-access-tq4kp\") pod \"insights-runtime-extractor-smgf6\" (UID: \"aeef13dc-e06e-49c5-b893-64c7f73a7e44\") " pod="openshift-insights/insights-runtime-extractor-smgf6"
Oct 30 19:16:15.320399 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.320357    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5338b461-6b7c-474b-8458-d1b53775ee0f-service-ca-bundle\") pod \"periodic-gathering-2lcwk-gjv8f\" (UID: \"5338b461-6b7c-474b-8458-d1b53775ee0f\") " pod="openshift-insights/periodic-gathering-2lcwk-gjv8f"
Oct 30 19:16:15.320460 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.320419    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5338b461-6b7c-474b-8458-d1b53775ee0f-trusted-ca-bundle\") pod \"periodic-gathering-2lcwk-gjv8f\" (UID: \"5338b461-6b7c-474b-8458-d1b53775ee0f\") " pod="openshift-insights/periodic-gathering-2lcwk-gjv8f"
Oct 30 19:16:15.320460 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.320455    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aeef13dc-e06e-49c5-b893-64c7f73a7e44-data-volume\") pod \"insights-runtime-extractor-smgf6\" (UID: \"aeef13dc-e06e-49c5-b893-64c7f73a7e44\") " pod="openshift-insights/insights-runtime-extractor-smgf6"
Oct 30 19:16:15.320801 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.320781    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aeef13dc-e06e-49c5-b893-64c7f73a7e44-data-volume\") pod \"insights-runtime-extractor-smgf6\" (UID: \"aeef13dc-e06e-49c5-b893-64c7f73a7e44\") " pod="openshift-insights/insights-runtime-extractor-smgf6"
Oct 30 19:16:15.320880 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.320865    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aeef13dc-e06e-49c5-b893-64c7f73a7e44-crio-socket\") pod \"insights-runtime-extractor-smgf6\" (UID: \"aeef13dc-e06e-49c5-b893-64c7f73a7e44\") " pod="openshift-insights/insights-runtime-extractor-smgf6"
Oct 30 19:16:15.320973 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:15.320960    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 19:16:15.321036 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:15.321027    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aeef13dc-e06e-49c5-b893-64c7f73a7e44-kube-rbac-proxy-cm podName:aeef13dc-e06e-49c5-b893-64c7f73a7e44 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:15.821008207 +0000 UTC m=+37.629189210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/aeef13dc-e06e-49c5-b893-64c7f73a7e44-kube-rbac-proxy-cm") pod "insights-runtime-extractor-smgf6" (UID: "aeef13dc-e06e-49c5-b893-64c7f73a7e44") : configmap "kube-rbac-proxy" not found
Oct 30 19:16:15.321310 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:15.321298    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:16:15.321400 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:15.321390    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeef13dc-e06e-49c5-b893-64c7f73a7e44-insights-runtime-extractor-tls podName:aeef13dc-e06e-49c5-b893-64c7f73a7e44 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:15.821375247 +0000 UTC m=+37.629556246 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aeef13dc-e06e-49c5-b893-64c7f73a7e44-insights-runtime-extractor-tls") pod "insights-runtime-extractor-smgf6" (UID: "aeef13dc-e06e-49c5-b893-64c7f73a7e44") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:16:15.329761 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.329721    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq4kp\" (UniqueName: \"kubernetes.io/projected/aeef13dc-e06e-49c5-b893-64c7f73a7e44-kube-api-access-tq4kp\") pod \"insights-runtime-extractor-smgf6\" (UID: \"aeef13dc-e06e-49c5-b893-64c7f73a7e44\") " pod="openshift-insights/insights-runtime-extractor-smgf6"
Oct 30 19:16:15.421508 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.421419    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5338b461-6b7c-474b-8458-d1b53775ee0f-archives-path\") pod \"periodic-gathering-2lcwk-gjv8f\" (UID: \"5338b461-6b7c-474b-8458-d1b53775ee0f\") " pod="openshift-insights/periodic-gathering-2lcwk-gjv8f"
Oct 30 19:16:15.421667 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.421516    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5338b461-6b7c-474b-8458-d1b53775ee0f-service-ca-bundle\") pod \"periodic-gathering-2lcwk-gjv8f\" (UID: \"5338b461-6b7c-474b-8458-d1b53775ee0f\") " pod="openshift-insights/periodic-gathering-2lcwk-gjv8f"
Oct 30 19:16:15.421667 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.421544    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5338b461-6b7c-474b-8458-d1b53775ee0f-trusted-ca-bundle\") pod \"periodic-gathering-2lcwk-gjv8f\" (UID: \"5338b461-6b7c-474b-8458-d1b53775ee0f\") " pod="openshift-insights/periodic-gathering-2lcwk-gjv8f"
Oct 30 19:16:15.421832 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.421724    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xrrr\" (UniqueName: \"kubernetes.io/projected/5338b461-6b7c-474b-8458-d1b53775ee0f-kube-api-access-4xrrr\") pod \"periodic-gathering-2lcwk-gjv8f\" (UID: \"5338b461-6b7c-474b-8458-d1b53775ee0f\") " pod="openshift-insights/periodic-gathering-2lcwk-gjv8f"
Oct 30 19:16:15.421892 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.421877    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5338b461-6b7c-474b-8458-d1b53775ee0f-archives-path\") pod \"periodic-gathering-2lcwk-gjv8f\" (UID: \"5338b461-6b7c-474b-8458-d1b53775ee0f\") " pod="openshift-insights/periodic-gathering-2lcwk-gjv8f"
Oct 30 19:16:15.422600 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.422577    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5338b461-6b7c-474b-8458-d1b53775ee0f-trusted-ca-bundle\") pod \"periodic-gathering-2lcwk-gjv8f\" (UID: \"5338b461-6b7c-474b-8458-d1b53775ee0f\") " pod="openshift-insights/periodic-gathering-2lcwk-gjv8f"
Oct 30 19:16:15.423131 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.423112    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5338b461-6b7c-474b-8458-d1b53775ee0f-service-ca-bundle\") pod \"periodic-gathering-2lcwk-gjv8f\" (UID: \"5338b461-6b7c-474b-8458-d1b53775ee0f\") " pod="openshift-insights/periodic-gathering-2lcwk-gjv8f"
Oct 30 19:16:15.435357 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.435325    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xrrr\" (UniqueName: \"kubernetes.io/projected/5338b461-6b7c-474b-8458-d1b53775ee0f-kube-api-access-4xrrr\") pod \"periodic-gathering-2lcwk-gjv8f\" (UID: \"5338b461-6b7c-474b-8458-d1b53775ee0f\") " pod="openshift-insights/periodic-gathering-2lcwk-gjv8f"
Oct 30 19:16:15.486954 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.486919    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vv4df"]
Oct 30 19:16:15.524451 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.522430    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-qggf8"]
Oct 30 19:16:15.524451 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.524428    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vv4df"
Oct 30 19:16:15.538779 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.531072    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 19:16:15.538779 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.531241    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 19:16:15.538779 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.531362    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-fppr2\""
Oct 30 19:16:15.538779 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.531750    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-2lcwk-gjv8f"
Oct 30 19:16:15.546806 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.546692    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vv4df"]
Oct 30 19:16:15.546806 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.546730    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-qggf8"]
Oct 30 19:16:15.546979 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.546834    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qggf8"
Oct 30 19:16:15.625167 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.625057    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hx4s\" (UniqueName: \"kubernetes.io/projected/853a64ec-147d-42eb-8946-e335de7878d3-kube-api-access-7hx4s\") pod \"migrator-66df5d496b-qggf8\" (UID: \"853a64ec-147d-42eb-8946-e335de7878d3\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qggf8"
Oct 30 19:16:15.625284 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.625226    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr447\" (UniqueName: \"kubernetes.io/projected/16705cc4-1ecc-47d1-9f4a-bb7eadc6294c-kube-api-access-qr447\") pod \"migrator-66df5d496b-vv4df\" (UID: \"16705cc4-1ecc-47d1-9f4a-bb7eadc6294c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vv4df"
Oct 30 19:16:15.726663 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.726623    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr447\" (UniqueName: \"kubernetes.io/projected/16705cc4-1ecc-47d1-9f4a-bb7eadc6294c-kube-api-access-qr447\") pod \"migrator-66df5d496b-vv4df\" (UID: \"16705cc4-1ecc-47d1-9f4a-bb7eadc6294c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vv4df"
Oct 30 19:16:15.726960 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.726862    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hx4s\" (UniqueName: \"kubernetes.io/projected/853a64ec-147d-42eb-8946-e335de7878d3-kube-api-access-7hx4s\") pod \"migrator-66df5d496b-qggf8\" (UID: \"853a64ec-147d-42eb-8946-e335de7878d3\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qggf8"
Oct 30 19:16:15.735184 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.735119    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hx4s\" (UniqueName: \"kubernetes.io/projected/853a64ec-147d-42eb-8946-e335de7878d3-kube-api-access-7hx4s\") pod \"migrator-66df5d496b-qggf8\" (UID: \"853a64ec-147d-42eb-8946-e335de7878d3\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qggf8"
Oct 30 19:16:15.736502 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.736474    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr447\" (UniqueName: \"kubernetes.io/projected/16705cc4-1ecc-47d1-9f4a-bb7eadc6294c-kube-api-access-qr447\") pod \"migrator-66df5d496b-vv4df\" (UID: \"16705cc4-1ecc-47d1-9f4a-bb7eadc6294c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vv4df"
Oct 30 19:16:15.827978 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.827922    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aeef13dc-e06e-49c5-b893-64c7f73a7e44-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-smgf6\" (UID: \"aeef13dc-e06e-49c5-b893-64c7f73a7e44\") " pod="openshift-insights/insights-runtime-extractor-smgf6"
Oct 30 19:16:15.827978 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.827968    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aeef13dc-e06e-49c5-b893-64c7f73a7e44-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-smgf6\" (UID: \"aeef13dc-e06e-49c5-b893-64c7f73a7e44\") " pod="openshift-insights/insights-runtime-extractor-smgf6"
Oct 30 19:16:15.828209 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:15.828065    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 19:16:15.828209 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:15.828100    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:16:15.828209 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:15.828141    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aeef13dc-e06e-49c5-b893-64c7f73a7e44-kube-rbac-proxy-cm podName:aeef13dc-e06e-49c5-b893-64c7f73a7e44 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:16.828121699 +0000 UTC m=+38.636302707 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/aeef13dc-e06e-49c5-b893-64c7f73a7e44-kube-rbac-proxy-cm") pod "insights-runtime-extractor-smgf6" (UID: "aeef13dc-e06e-49c5-b893-64c7f73a7e44") : configmap "kube-rbac-proxy" not found
Oct 30 19:16:15.828209 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:15.828161    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeef13dc-e06e-49c5-b893-64c7f73a7e44-insights-runtime-extractor-tls podName:aeef13dc-e06e-49c5-b893-64c7f73a7e44 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:16.828151936 +0000 UTC m=+38.636332936 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aeef13dc-e06e-49c5-b893-64c7f73a7e44-insights-runtime-extractor-tls") pod "insights-runtime-extractor-smgf6" (UID: "aeef13dc-e06e-49c5-b893-64c7f73a7e44") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:16:15.841645 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.841615    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-2lcwk-gjv8f"]
Oct 30 19:16:15.846058 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.846036    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-b8cs2"]
Oct 30 19:16:15.849388 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:16:15.849309    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5338b461_6b7c_474b_8458_d1b53775ee0f.slice/crio-487fa79e6ceaed2223a66678071091aae7f373b853288eb4e2de23616b76a96b WatchSource:0}: Error finding container 487fa79e6ceaed2223a66678071091aae7f373b853288eb4e2de23616b76a96b: Status 404 returned error can't find the container with id 487fa79e6ceaed2223a66678071091aae7f373b853288eb4e2de23616b76a96b
Oct 30 19:16:15.855616 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.855463    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vv4df"
Oct 30 19:16:15.879037 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.878977    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-b8cs2"]
Oct 30 19:16:15.879143 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.879100    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-b8cs2"
Oct 30 19:16:15.882706 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.882687    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 19:16:15.882850 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.882833    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-46v7v\""
Oct 30 19:16:15.882942 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.882930    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 19:16:15.883007 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.882977    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 19:16:15.883074 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.883063    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 19:16:15.889510 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.889483    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qggf8"
Oct 30 19:16:15.929182 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.929158    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vk66\" (UniqueName: \"kubernetes.io/projected/a37bca55-21cf-4b76-9c05-e2c951647bf8-kube-api-access-2vk66\") pod \"service-ca-5fffc97bdf-b8cs2\" (UID: \"a37bca55-21cf-4b76-9c05-e2c951647bf8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-b8cs2"
Oct 30 19:16:15.929298 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.929261    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a37bca55-21cf-4b76-9c05-e2c951647bf8-signing-key\") pod \"service-ca-5fffc97bdf-b8cs2\" (UID: \"a37bca55-21cf-4b76-9c05-e2c951647bf8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-b8cs2"
Oct 30 19:16:15.929298 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.929291    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a37bca55-21cf-4b76-9c05-e2c951647bf8-signing-cabundle\") pod \"service-ca-5fffc97bdf-b8cs2\" (UID: \"a37bca55-21cf-4b76-9c05-e2c951647bf8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-b8cs2"
Oct 30 19:16:15.983930 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.983313    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bvcb4" event={"ID":"2d52a94d-72ac-4e5c-9472-104399a21a3b","Type":"ContainerStarted","Data":"98771dbae0b04941ad08e283ddbf6bc26d1c31fae97a38d48b73cecd69e2c814"}
Oct 30 19:16:15.992900 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.991861    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cmw6w_76e21916-0744-4499-a0cb-64a7a44a8285/console-operator/1.log"
Oct 30 19:16:15.992900 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.992349    2579 scope.go:122] "RemoveContainer" containerID="b3f38d74f7964cb2e0db345a5bf20e6f2b18ed1ffffa37aafe2a406f3f91103c"
Oct 30 19:16:15.992900 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:15.992558    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-cmw6w_openshift-console-operator(76e21916-0744-4499-a0cb-64a7a44a8285)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-cmw6w" podUID="76e21916-0744-4499-a0cb-64a7a44a8285"
Oct 30 19:16:15.995903 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:15.995876    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-2lcwk-gjv8f" event={"ID":"5338b461-6b7c-474b-8458-d1b53775ee0f","Type":"ContainerStarted","Data":"487fa79e6ceaed2223a66678071091aae7f373b853288eb4e2de23616b76a96b"}
Oct 30 19:16:16.008664 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:16.006529    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bvcb4" podStartSLOduration=387.159713553 podStartE2EDuration="6m33.006512849s" totalImagesPullingTime="5.846799296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:16.005301354 +0000 UTC m=+37.813482366" watchObservedRunningTime="2026-10-30 19:16:16.006512849 +0000 UTC m=+37.814693866"
Oct 30 19:16:16.031678 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:16.031647    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vk66\" (UniqueName: \"kubernetes.io/projected/a37bca55-21cf-4b76-9c05-e2c951647bf8-kube-api-access-2vk66\") pod \"service-ca-5fffc97bdf-b8cs2\" (UID: \"a37bca55-21cf-4b76-9c05-e2c951647bf8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-b8cs2"
Oct 30 19:16:16.032081 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:16.032048    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a37bca55-21cf-4b76-9c05-e2c951647bf8-signing-key\") pod \"service-ca-5fffc97bdf-b8cs2\" (UID: \"a37bca55-21cf-4b76-9c05-e2c951647bf8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-b8cs2"
Oct 30 19:16:16.032154 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:16.032111    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a37bca55-21cf-4b76-9c05-e2c951647bf8-signing-cabundle\") pod \"service-ca-5fffc97bdf-b8cs2\" (UID: \"a37bca55-21cf-4b76-9c05-e2c951647bf8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-b8cs2"
Oct 30 19:16:16.034154 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:16.034130    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a37bca55-21cf-4b76-9c05-e2c951647bf8-signing-cabundle\") pod \"service-ca-5fffc97bdf-b8cs2\" (UID: \"a37bca55-21cf-4b76-9c05-e2c951647bf8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-b8cs2"
Oct 30 19:16:16.037721 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:16.037697    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a37bca55-21cf-4b76-9c05-e2c951647bf8-signing-key\") pod \"service-ca-5fffc97bdf-b8cs2\" (UID: \"a37bca55-21cf-4b76-9c05-e2c951647bf8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-b8cs2"
Oct 30 19:16:16.041671 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:16.041653    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vk66\" (UniqueName: \"kubernetes.io/projected/a37bca55-21cf-4b76-9c05-e2c951647bf8-kube-api-access-2vk66\") pod \"service-ca-5fffc97bdf-b8cs2\" (UID: \"a37bca55-21cf-4b76-9c05-e2c951647bf8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-b8cs2"
Oct 30 19:16:16.186135 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:16.186012    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vv4df"]
Oct 30 19:16:16.189652 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:16:16.189618    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16705cc4_1ecc_47d1_9f4a_bb7eadc6294c.slice/crio-1b81426a4a1dfc6148525fe704c7fc993cfeb9bd6a25c91c4ba435fbafe15bce WatchSource:0}: Error finding container 1b81426a4a1dfc6148525fe704c7fc993cfeb9bd6a25c91c4ba435fbafe15bce: Status 404 returned error can't find the container with id 1b81426a4a1dfc6148525fe704c7fc993cfeb9bd6a25c91c4ba435fbafe15bce
Oct 30 19:16:16.213274 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:16.213242    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-qggf8"]
Oct 30 19:16:16.216084 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:16:16.216058    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod853a64ec_147d_42eb_8946_e335de7878d3.slice/crio-103e94309533e5b3f462ba5781ecee4e8885c5ccf33c62bf1f560d1f596b95df WatchSource:0}: Error finding container 103e94309533e5b3f462ba5781ecee4e8885c5ccf33c62bf1f560d1f596b95df: Status 404 returned error can't find the container with id 103e94309533e5b3f462ba5781ecee4e8885c5ccf33c62bf1f560d1f596b95df
Oct 30 19:16:16.224393 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:16.224369    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-b8cs2"
Oct 30 19:16:16.436889 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:16.436851    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/247ff425-bed3-4297-b5c0-cc32b21bc179-metrics-certs\") pod \"router-default-75866dc98c-4mnhf\" (UID: \"247ff425-bed3-4297-b5c0-cc32b21bc179\") " pod="openshift-ingress/router-default-75866dc98c-4mnhf"
Oct 30 19:16:16.437071 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:16.436932    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a85cca66-a438-4117-9f50-977c7f528b7c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lpg7w\" (UID: \"a85cca66-a438-4117-9f50-977c7f528b7c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lpg7w"
Oct 30 19:16:16.437071 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:16.436976    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0269ffa6-ce06-43f3-be2a-58f6aa3e6964-metrics-tls\") pod \"dns-default-gkn6q\" (UID: \"0269ffa6-ce06-43f3-be2a-58f6aa3e6964\") " pod="openshift-dns/dns-default-gkn6q"
Oct 30 19:16:16.437071 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:16.437022    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/06397fd4-3377-4eaa-8879-ebbb5943fbda-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2sdkw\" (UID: \"06397fd4-3377-4eaa-8879-ebbb5943fbda\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2sdkw"
Oct 30 19:16:16.437071 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:16.437049    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-registry-tls\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:16.437261 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:16.437072    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/247ff425-bed3-4297-b5c0-cc32b21bc179-service-ca-bundle\") pod \"router-default-75866dc98c-4mnhf\" (UID: \"247ff425-bed3-4297-b5c0-cc32b21bc179\") " pod="openshift-ingress/router-default-75866dc98c-4mnhf"
Oct 30 19:16:16.437261 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:16.437140    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cbc4f7c7-bebd-4200-833e-1afbf3c2c81a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-p5nzf\" (UID: \"cbc4f7c7-bebd-4200-833e-1afbf3c2c81a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p5nzf"
Oct 30 19:16:16.437261 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:16.437184    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-registry-tls\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:16.437402 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:16.437313    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:16:16.437402 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:16.437337    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-748f8689f4-6lwjn: secret "image-registry-tls" not found
Oct 30 19:16:16.437497 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:16.437407    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-registry-tls podName:9f302411-01be-438c-a30c-70c944afd630 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:24.437389098 +0000 UTC m=+46.245570106 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-registry-tls") pod "image-registry-748f8689f4-6lwjn" (UID: "9f302411-01be-438c-a30c-70c944afd630") : secret "image-registry-tls" not found
Oct 30 19:16:16.437497 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:16.437481    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:16:16.437601 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:16.437518    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/247ff425-bed3-4297-b5c0-cc32b21bc179-metrics-certs podName:247ff425-bed3-4297-b5c0-cc32b21bc179 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:24.437507446 +0000 UTC m=+46.245688461 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/247ff425-bed3-4297-b5c0-cc32b21bc179-metrics-certs") pod "router-default-75866dc98c-4mnhf" (UID: "247ff425-bed3-4297-b5c0-cc32b21bc179") : secret "router-metrics-certs-default" not found
Oct 30 19:16:16.437601 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:16.437575    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:16:16.437701 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:16.437609    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a85cca66-a438-4117-9f50-977c7f528b7c-networking-console-plugin-cert podName:a85cca66-a438-4117-9f50-977c7f528b7c nodeName:}" failed. No retries permitted until 2026-10-30 19:16:24.437597723 +0000 UTC m=+46.245778718 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a85cca66-a438-4117-9f50-977c7f528b7c-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-lpg7w" (UID: "a85cca66-a438-4117-9f50-977c7f528b7c") : secret "networking-console-plugin-cert" not found
Oct 30 19:16:16.437701 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:16.437665    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:16:16.437701 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:16.437695    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0269ffa6-ce06-43f3-be2a-58f6aa3e6964-metrics-tls podName:0269ffa6-ce06-43f3-be2a-58f6aa3e6964 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:24.437685715 +0000 UTC m=+46.245866715 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0269ffa6-ce06-43f3-be2a-58f6aa3e6964-metrics-tls") pod "dns-default-gkn6q" (UID: "0269ffa6-ce06-43f3-be2a-58f6aa3e6964") : secret "dns-default-metrics-tls" not found
Oct 30 19:16:16.437907 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:16.437767    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:16:16.437907 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:16.437804    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06397fd4-3377-4eaa-8879-ebbb5943fbda-cluster-monitoring-operator-tls podName:06397fd4-3377-4eaa-8879-ebbb5943fbda nodeName:}" failed. No retries permitted until 2026-10-30 19:16:24.437792023 +0000 UTC m=+46.245973019 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/06397fd4-3377-4eaa-8879-ebbb5943fbda-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2sdkw" (UID: "06397fd4-3377-4eaa-8879-ebbb5943fbda") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:16:16.437907 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:16.437861    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:16:16.437907 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:16.437872    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78dc458d47-tqn8s: secret "image-registry-tls" not found
Oct 30 19:16:16.437907 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:16.437901    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-registry-tls podName:a64211d0-1c48-41d2-b254-4ae84d5097d7 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:24.437890879 +0000 UTC m=+46.246071877 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-registry-tls") pod "image-registry-78dc458d47-tqn8s" (UID: "a64211d0-1c48-41d2-b254-4ae84d5097d7") : secret "image-registry-tls" not found
Oct 30 19:16:16.438144 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:16.437965    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/247ff425-bed3-4297-b5c0-cc32b21bc179-service-ca-bundle podName:247ff425-bed3-4297-b5c0-cc32b21bc179 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:24.437955585 +0000 UTC m=+46.246136580 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/247ff425-bed3-4297-b5c0-cc32b21bc179-service-ca-bundle") pod "router-default-75866dc98c-4mnhf" (UID: "247ff425-bed3-4297-b5c0-cc32b21bc179") : configmap references non-existent config key: service-ca.crt
Oct 30 19:16:16.438144 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:16.438022    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 19:16:16.438144 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:16.438052    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbc4f7c7-bebd-4200-833e-1afbf3c2c81a-samples-operator-tls podName:cbc4f7c7-bebd-4200-833e-1afbf3c2c81a nodeName:}" failed. No retries permitted until 2026-10-30 19:16:24.438042505 +0000 UTC m=+46.246223502 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cbc4f7c7-bebd-4200-833e-1afbf3c2c81a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-p5nzf" (UID: "cbc4f7c7-bebd-4200-833e-1afbf3c2c81a") : secret "samples-operator-tls" not found
Oct 30 19:16:16.539166 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:16.539137    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-b8cs2"]
Oct 30 19:16:16.541557 ip-10-0-137-84 kubenswrapper[2579]: W1030 19:16:16.541524    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda37bca55_21cf_4b76_9c05_e2c951647bf8.slice/crio-2268a2c217d2bf4d82f1c06fefc397fff88c6c383fddbdaa5c240ca4a9fdde56 WatchSource:0}: Error finding container 2268a2c217d2bf4d82f1c06fefc397fff88c6c383fddbdaa5c240ca4a9fdde56: Status 404 returned error can't find the container with id 2268a2c217d2bf4d82f1c06fefc397fff88c6c383fddbdaa5c240ca4a9fdde56
Oct 30 19:16:16.841605 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:16.841524    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aeef13dc-e06e-49c5-b893-64c7f73a7e44-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-smgf6\" (UID: \"aeef13dc-e06e-49c5-b893-64c7f73a7e44\") " pod="openshift-insights/insights-runtime-extractor-smgf6"
Oct 30 19:16:16.841605 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:16.841562    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aeef13dc-e06e-49c5-b893-64c7f73a7e44-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-smgf6\" (UID: \"aeef13dc-e06e-49c5-b893-64c7f73a7e44\") " pod="openshift-insights/insights-runtime-extractor-smgf6"
Oct 30 19:16:16.841837 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:16.841682    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:16:16.841837 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:16.841756    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeef13dc-e06e-49c5-b893-64c7f73a7e44-insights-runtime-extractor-tls podName:aeef13dc-e06e-49c5-b893-64c7f73a7e44 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:18.841720569 +0000 UTC m=+40.649901591 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aeef13dc-e06e-49c5-b893-64c7f73a7e44-insights-runtime-extractor-tls") pod "insights-runtime-extractor-smgf6" (UID: "aeef13dc-e06e-49c5-b893-64c7f73a7e44") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:16:16.842142 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:16.842128    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 19:16:16.842184 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:16.842171    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aeef13dc-e06e-49c5-b893-64c7f73a7e44-kube-rbac-proxy-cm podName:aeef13dc-e06e-49c5-b893-64c7f73a7e44 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:18.842157564 +0000 UTC m=+40.650338558 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/aeef13dc-e06e-49c5-b893-64c7f73a7e44-kube-rbac-proxy-cm") pod "insights-runtime-extractor-smgf6" (UID: "aeef13dc-e06e-49c5-b893-64c7f73a7e44") : configmap "kube-rbac-proxy" not found
Oct 30 19:16:17.003527 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:17.003479    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qggf8" event={"ID":"853a64ec-147d-42eb-8946-e335de7878d3","Type":"ContainerStarted","Data":"103e94309533e5b3f462ba5781ecee4e8885c5ccf33c62bf1f560d1f596b95df"}
Oct 30 19:16:17.004966 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:17.004943    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vv4df" event={"ID":"16705cc4-1ecc-47d1-9f4a-bb7eadc6294c","Type":"ContainerStarted","Data":"1b81426a4a1dfc6148525fe704c7fc993cfeb9bd6a25c91c4ba435fbafe15bce"}
Oct 30 19:16:17.007332 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:17.007250    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-2lcwk-gjv8f" event={"ID":"5338b461-6b7c-474b-8458-d1b53775ee0f","Type":"ContainerStarted","Data":"6081dbd078e020eb140d07eec8006a13c5f61e5409c51c7e49470b00cd9133f6"}
Oct 30 19:16:17.010185 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:17.010098    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-b8cs2" event={"ID":"a37bca55-21cf-4b76-9c05-e2c951647bf8","Type":"ContainerStarted","Data":"392925e1b44580796adaf6256863498fec8d5bf383e9c4b816315b4df119f17b"}
Oct 30 19:16:17.010185 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:17.010144    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-b8cs2" event={"ID":"a37bca55-21cf-4b76-9c05-e2c951647bf8","Type":"ContainerStarted","Data":"2268a2c217d2bf4d82f1c06fefc397fff88c6c383fddbdaa5c240ca4a9fdde56"}
Oct 30 19:16:17.025327 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:17.024368    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-2lcwk-gjv8f" podStartSLOduration=2.024350089 podStartE2EDuration="2.024350089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:17.022696795 +0000 UTC m=+38.830877833" watchObservedRunningTime="2026-10-30 19:16:17.024350089 +0000 UTC m=+38.832531107"
Oct 30 19:16:17.043802 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:17.043749    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-b8cs2" podStartSLOduration=2.043717442 podStartE2EDuration="2.043717442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:17.04211511 +0000 UTC m=+38.850296127" watchObservedRunningTime="2026-10-30 19:16:17.043717442 +0000 UTC m=+38.851898460"
Oct 30 19:16:18.014339 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:18.014303    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-n9zx2" event={"ID":"47e9832e-bc96-445a-a93e-9d24f84db3b3","Type":"ContainerStarted","Data":"33f3b6c9744a81831aaebf3d2be34fec262565413a6794e4b5614b61551c2b8b"}
Oct 30 19:16:18.014726 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:18.014672    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-n9zx2"
Oct 30 19:16:18.032338 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:18.032292    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-n9zx2" podStartSLOduration=35.078248777 podStartE2EDuration="39.032281091s" totalImagesPullingTime="3.954032314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:18.031430007 +0000 UTC m=+39.839611037" watchObservedRunningTime="2026-10-30 19:16:18.032281091 +0000 UTC m=+39.840462106"
Oct 30 19:16:18.390236 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:18.390208    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xjjqh_36ab52a2-65e2-4af6-a872-9052e12714d1/dns-node-resolver/0.log"
Oct 30 19:16:18.866678 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:18.866641    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aeef13dc-e06e-49c5-b893-64c7f73a7e44-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-smgf6\" (UID: \"aeef13dc-e06e-49c5-b893-64c7f73a7e44\") " pod="openshift-insights/insights-runtime-extractor-smgf6"
Oct 30 19:16:18.866678 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:18.866680    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aeef13dc-e06e-49c5-b893-64c7f73a7e44-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-smgf6\" (UID: \"aeef13dc-e06e-49c5-b893-64c7f73a7e44\") " pod="openshift-insights/insights-runtime-extractor-smgf6"
Oct 30 19:16:18.866946 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:18.866807    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 19:16:18.866946 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:18.866857    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:16:18.866946 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:18.866903    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aeef13dc-e06e-49c5-b893-64c7f73a7e44-kube-rbac-proxy-cm podName:aeef13dc-e06e-49c5-b893-64c7f73a7e44 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:22.866881961 +0000 UTC m=+44.675062956 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/aeef13dc-e06e-49c5-b893-64c7f73a7e44-kube-rbac-proxy-cm") pod "insights-runtime-extractor-smgf6" (UID: "aeef13dc-e06e-49c5-b893-64c7f73a7e44") : configmap "kube-rbac-proxy" not found
Oct 30 19:16:18.866946 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:18.866926    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeef13dc-e06e-49c5-b893-64c7f73a7e44-insights-runtime-extractor-tls podName:aeef13dc-e06e-49c5-b893-64c7f73a7e44 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:22.866915639 +0000 UTC m=+44.675096644 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aeef13dc-e06e-49c5-b893-64c7f73a7e44-insights-runtime-extractor-tls") pod "insights-runtime-extractor-smgf6" (UID: "aeef13dc-e06e-49c5-b893-64c7f73a7e44") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:16:18.900213 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:18.900184    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-cmw6w"
Oct 30 19:16:18.900377 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:18.900241    2579 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-cmw6w"
Oct 30 19:16:18.900655 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:18.900637    2579 scope.go:122] "RemoveContainer" containerID="b3f38d74f7964cb2e0db345a5bf20e6f2b18ed1ffffa37aafe2a406f3f91103c"
Oct 30 19:16:18.900921 ip-10-0-137-84 kubenswrapper[2579]: E1030 19:16:18.900889    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-cmw6w_openshift-console-operator(76e21916-0744-4499-a0cb-64a7a44a8285)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-cmw6w" podUID="76e21916-0744-4499-a0cb-64a7a44a8285"
Oct 30 19:16:19.020699 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:19.020660    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qggf8" event={"ID":"853a64ec-147d-42eb-8946-e335de7878d3","Type":"ContainerStarted","Data":"8755aa69adea916097bd53d81dbf1273150e138f8407b9b1f16541ff7f610fc3"}
Oct 30 19:16:19.022541 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:19.022511    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vv4df" event={"ID":"16705cc4-1ecc-47d1-9f4a-bb7eadc6294c","Type":"ContainerStarted","Data":"01615120ee011681c9ecabb8f11c9f93f4c8e44df2266b4ecf8d63975694fcb2"}
Oct 30 19:16:19.038659 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:19.038607    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qggf8" podStartSLOduration=1.962654044 podStartE2EDuration="4.038591424s" totalImagesPullingTime="2.07593738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:19.037198183 +0000 UTC m=+40.845379200" watchObservedRunningTime="2026-10-30 19:16:19.038591424 +0000 UTC m=+40.846772442"
Oct 30 19:16:19.053991 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:19.053945    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vv4df" podStartSLOduration=2.196115992 podStartE2EDuration="4.053928169s" totalImagesPullingTime="1.857812177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:19.052103828 +0000 UTC m=+40.860284845" watchObservedRunningTime="2026-10-30 19:16:19.053928169 +0000 UTC m=+40.862109184"
Oct 30 19:16:19.389632 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:19.389597    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zj8d6_f1ea27b2-80df-404c-ac39-460f495957b2/node-ca/0.log"
Oct 30 19:16:19.789234 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:19.789181    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9jkcb_765d8d9d-b89f-419f-b237-3f8034db1bed/serve-healthcheck-canary/0.log"
Oct 30 19:16:19.990631 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:19.990592    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-qggf8_853a64ec-147d-42eb-8946-e335de7878d3/migrator/0.log"
Oct 30 19:16:20.189830 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:20.189795    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-vv4df_16705cc4-1ecc-47d1-9f4a-bb7eadc6294c/migrator/0.log"
Oct 30 19:16:20.391897 ip-10-0-137-84 kubenswrapper[2579]: I1030 19:16:20.391809    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-62bhw_353ebfb4-7e11-41de-b992-277443f56960/kube-storage-version-migrator-operator/0.log"
Oct 30 19:16:20.426320 ip-10-0-137-84 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 19:16:20.436358 ip-10-0-137-84 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 19:16:20.436597 ip-10-0-137-84 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 19:16:20.436653 ip-10-0-137-84 systemd[1]: kubelet.service: Consumed 3.233s CPU time.
Oct 30 19:16:20.457276 ip-10-0-137-84 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 19:16:20.529591 ip-10-0-137-84 kubenswrapper[5043]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:16:20.529591 ip-10-0-137-84 kubenswrapper[5043]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 19:16:20.529591 ip-10-0-137-84 kubenswrapper[5043]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:16:20.529591 ip-10-0-137-84 kubenswrapper[5043]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:16:20.531915 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.531833    5043 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:16:20.531915 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.531915    5043 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:16:20.531992 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.531919    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:16:20.531992 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.531923    5043 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:16:20.531992 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.531927    5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:16:20.531992 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.531930    5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:16:20.531992 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.531933    5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:16:20.531992 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.531935    5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:16:20.531992 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.531938    5043 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:16:20.531992 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.531942    5043 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:16:20.531992 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.531946    5043 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:16:20.531992 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.531949    5043 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:16:20.531992 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.531951    5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:16:20.531992 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.531954    5043 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:16:20.531992 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.531956    5043 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:16:20.531992 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.531959    5043 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:16:20.531992 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.531961    5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:16:20.531992 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.531964    5043 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:16:20.531992 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.531967    5043 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:16:20.531992 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.531970    5043 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:16:20.531992 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.531972    5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:16:20.532448 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.531974    5043 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:16:20.532448 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.531977    5043 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:16:20.532448 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.531980    5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:16:20.532448 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.531982    5043 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:16:20.532448 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.531985    5043 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:16:20.532448 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.531995    5043 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:16:20.532448 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.531998    5043 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:16:20.532448 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532000    5043 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:16:20.532448 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532003    5043 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:16:20.532448 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532006    5043 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:16:20.532448 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532008    5043 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:16:20.532448 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532011    5043 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:16:20.532448 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532013    5043 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:16:20.532448 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532016    5043 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:16:20.532448 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532019    5043 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:16:20.532448 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532022    5043 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:16:20.532448 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532024    5043 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:16:20.532448 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532026    5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:16:20.532448 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532029    5043 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:16:20.532448 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532031    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:16:20.532951 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532034    5043 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:16:20.532951 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532036    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:16:20.532951 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532039    5043 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:16:20.532951 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532041    5043 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:16:20.532951 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532044    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:16:20.532951 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532047    5043 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:16:20.532951 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532050    5043 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:16:20.532951 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532053    5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:16:20.532951 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532055    5043 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:16:20.532951 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532058    5043 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:16:20.532951 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532060    5043 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:16:20.532951 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532062    5043 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:16:20.532951 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532065    5043 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:16:20.532951 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532067    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:16:20.532951 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532070    5043 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:16:20.532951 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532073    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:16:20.532951 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532075    5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:16:20.532951 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532078    5043 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:16:20.532951 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532080    5043 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:16:20.532951 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532083    5043 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:16:20.533440 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532085    5043 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:16:20.533440 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532088    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:16:20.533440 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532090    5043 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:16:20.533440 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532092    5043 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:16:20.533440 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532095    5043 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:16:20.533440 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532097    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:16:20.533440 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532100    5043 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:16:20.533440 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532102    5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:16:20.533440 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532104    5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:16:20.533440 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532107    5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:16:20.533440 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532110    5043 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:16:20.533440 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532112    5043 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:16:20.533440 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532115    5043 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:16:20.533440 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532119    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:16:20.533440 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532124    5043 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:16:20.533440 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532127    5043 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:16:20.533440 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532132    5043 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:16:20.533440 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532135    5043 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:16:20.533440 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532148    5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:16:20.533927 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532151    5043 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:16:20.533927 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532154    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:16:20.533927 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532156    5043 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:16:20.533927 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532161    5043 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:16:20.533927 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532163    5043 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:16:20.533927 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532166    5043 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:16:20.533927 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532168    5043 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:16:20.533927 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532171    5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:16:20.533927 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532173    5043 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:16:20.533927 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532178    5043 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:16:20.533927 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532182    5043 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:16:20.533927 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532185    5043 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:16:20.533927 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532188    5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:16:20.533927 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532190    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:16:20.533927 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532193    5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:16:20.533927 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532195    5043 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:16:20.533927 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532198    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:16:20.533927 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532200    5043 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:16:20.533927 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532202    5043 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:16:20.534388 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532205    5043 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:16:20.534388 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532207    5043 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:16:20.534388 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532210    5043 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:16:20.534388 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532212    5043 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:16:20.534388 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532214    5043 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:16:20.534388 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532217    5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:16:20.534388 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.532219    5043 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:16:20.534388 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533033    5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:16:20.534388 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533040    5043 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:16:20.534388 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533044    5043 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:16:20.534388 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533047    5043 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:16:20.534388 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533050    5043 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:16:20.534388 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533053    5043 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:16:20.534388 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533056    5043 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:16:20.534388 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533059    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:16:20.534388 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533061    5043 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:16:20.534388 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533065    5043 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:16:20.534388 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533067    5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:16:20.534388 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533070    5043 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:16:20.534388 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533072    5043 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:16:20.534927 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533075    5043 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:16:20.534927 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533078    5043 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:16:20.534927 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533080    5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:16:20.534927 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533083    5043 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:16:20.534927 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533085    5043 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:16:20.534927 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533087    5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:16:20.534927 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533090    5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:16:20.534927 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533092    5043 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:16:20.534927 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533095    5043 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:16:20.534927 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533097    5043 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:16:20.534927 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533100    5043 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:16:20.534927 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533102    5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:16:20.534927 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533105    5043 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:16:20.534927 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533107    5043 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:16:20.534927 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533110    5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:16:20.534927 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533112    5043 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:16:20.534927 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533115    5043 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:16:20.534927 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533117    5043 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:16:20.534927 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533120    5043 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:16:20.535395 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533122    5043 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:16:20.535395 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533125    5043 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:16:20.535395 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533128    5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:16:20.535395 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533130    5043 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:16:20.535395 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533133    5043 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:16:20.535395 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533137    5043 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:16:20.535395 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533139    5043 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:16:20.535395 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533142    5043 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:16:20.535395 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533145    5043 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:16:20.535395 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533148    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:16:20.535395 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533150    5043 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:16:20.535395 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533153    5043 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:16:20.535395 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533155    5043 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:16:20.535395 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533158    5043 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:16:20.535395 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533161    5043 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:16:20.535395 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533163    5043 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:16:20.535395 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533166    5043 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:16:20.535395 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533169    5043 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:16:20.535395 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533171    5043 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:16:20.535395 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533174    5043 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:16:20.535918 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533176    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:16:20.535918 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533179    5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:16:20.535918 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533181    5043 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:16:20.535918 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533183    5043 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:16:20.535918 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533187    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:16:20.535918 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533189    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:16:20.535918 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533192    5043 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:16:20.535918 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533194    5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:16:20.535918 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533196    5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:16:20.535918 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533199    5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:16:20.535918 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533202    5043 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:16:20.535918 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533204    5043 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:16:20.535918 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533207    5043 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:16:20.535918 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533210    5043 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:16:20.535918 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533212    5043 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:16:20.535918 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533233    5043 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:16:20.535918 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533237    5043 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:16:20.535918 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533240    5043 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:16:20.535918 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533243    5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:16:20.535918 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533246    5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:16:20.536430 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533252    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:16:20.536430 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533255    5043 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:16:20.536430 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533258    5043 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:16:20.536430 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533260    5043 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:16:20.536430 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533263    5043 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:16:20.536430 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533266    5043 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:16:20.536430 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533270    5043 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:16:20.536430 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533273    5043 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:16:20.536430 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533276    5043 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:16:20.536430 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533279    5043 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:16:20.536430 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533282    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:16:20.536430 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533284    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:16:20.536430 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533287    5043 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:16:20.536430 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533289    5043 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:16:20.536430 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533292    5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:16:20.536430 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533294    5043 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:16:20.536430 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533297    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:16:20.536430 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533299    5043 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:16:20.536430 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533302    5043 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:16:20.536933 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533304    5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:16:20.536933 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533307    5043 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:16:20.536933 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533309    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:16:20.536933 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533313    5043 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:16:20.536933 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533317    5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:16:20.536933 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533320    5043 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:16:20.536933 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533331    5043 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:16:20.536933 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533334    5043 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:16:20.536933 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533336    5043 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:16:20.536933 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533339    5043 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:16:20.536933 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533342    5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:16:20.536933 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533345    5043 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:16:20.536933 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533347    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:16:20.536933 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533351    5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:16:20.536933 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533355    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:16:20.536933 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533908    5043 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:16:20.536933 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533914    5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:16:20.536933 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533917    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:16:20.536933 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533920    5043 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:16:20.537401 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533923    5043 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:16:20.537401 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533926    5043 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:16:20.537401 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533929    5043 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:16:20.537401 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533931    5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:16:20.537401 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533934    5043 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:16:20.537401 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533937    5043 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:16:20.537401 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533939    5043 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:16:20.537401 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533942    5043 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:16:20.537401 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533944    5043 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:16:20.537401 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533947    5043 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:16:20.537401 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533950    5043 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:16:20.537401 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533952    5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:16:20.537401 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533955    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:16:20.537401 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533957    5043 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:16:20.537401 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533960    5043 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:16:20.537401 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533962    5043 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:16:20.537401 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533965    5043 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:16:20.537401 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533968    5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:16:20.537401 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533970    5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:16:20.537875 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533972    5043 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:16:20.537875 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533975    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:16:20.537875 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533978    5043 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:16:20.537875 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533980    5043 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:16:20.537875 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533983    5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:16:20.537875 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533985    5043 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:16:20.537875 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533987    5043 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:16:20.537875 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.533992    5043 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:16:20.537875 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534001    5043 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:16:20.537875 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534003    5043 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:16:20.537875 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534006    5043 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:16:20.537875 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534009    5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:16:20.537875 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534011    5043 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:16:20.537875 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534014    5043 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:16:20.537875 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534016    5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:16:20.537875 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534018    5043 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:16:20.537875 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534021    5043 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:16:20.537875 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534023    5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:16:20.537875 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534026    5043 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:16:20.537875 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534028    5043 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:16:20.538362 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534031    5043 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:16:20.538362 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534033    5043 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:16:20.538362 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534035    5043 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:16:20.538362 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534038    5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:16:20.538362 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534040    5043 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:16:20.538362 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534043    5043 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:16:20.538362 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534045    5043 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:16:20.538362 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534048    5043 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:16:20.538362 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534050    5043 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:16:20.538362 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534053    5043 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:16:20.538362 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534055    5043 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:16:20.538362 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534058    5043 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:16:20.538362 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534060    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:16:20.538362 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534063    5043 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:16:20.538362 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534065    5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:16:20.538362 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534068    5043 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:16:20.538362 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534070    5043 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:16:20.538362 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534073    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:16:20.538362 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534075    5043 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:16:20.538865 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534080    5043 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:16:20.538865 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534084    5043 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:16:20.538865 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534092    5043 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:16:20.538865 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534097    5043 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:16:20.538865 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534099    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:16:20.538865 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534102    5043 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:16:20.538865 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534104    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:16:20.538865 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534107    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:16:20.538865 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534109    5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:16:20.538865 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534112    5043 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:16:20.538865 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534115    5043 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:16:20.538865 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534119    5043 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:16:20.538865 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534122    5043 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:16:20.538865 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534124    5043 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:16:20.538865 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534127    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:16:20.538865 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534130    5043 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:16:20.538865 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534132    5043 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:16:20.538865 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534135    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:16:20.538865 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534137    5043 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:16:20.539318 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534140    5043 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:16:20.539318 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534142    5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:16:20.539318 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534145    5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:16:20.539318 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534147    5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:16:20.539318 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534149    5043 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:16:20.539318 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534152    5043 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:16:20.539318 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534154    5043 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:16:20.539318 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534157    5043 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:16:20.539318 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534159    5043 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:16:20.539318 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534162    5043 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:16:20.539318 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534164    5043 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:16:20.539318 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534167    5043 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:16:20.539318 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534169    5043 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:16:20.539318 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534174    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:16:20.539318 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534176    5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:16:20.539318 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534180    5043 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:16:20.539318 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534188    5043 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:16:20.539318 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534190    5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:16:20.539318 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534193    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:16:20.539318 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534195    5043 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:16:20.539812 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534198    5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:16:20.539812 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534200    5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:16:20.539812 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534202    5043 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:16:20.539812 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534205    5043 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:16:20.539812 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.534207    5043 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:16:20.539812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534313    5043 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 19:16:20.539812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534323    5043 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 19:16:20.539812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534337    5043 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 19:16:20.539812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534341    5043 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 19:16:20.539812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534347    5043 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 19:16:20.539812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534350    5043 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 19:16:20.539812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534354    5043 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 19:16:20.539812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534358    5043 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 19:16:20.539812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534361    5043 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 19:16:20.539812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534364    5043 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 19:16:20.539812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534367    5043 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 19:16:20.539812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534371    5043 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 19:16:20.539812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534374    5043 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 19:16:20.539812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534377    5043 flags.go:64] FLAG: --cgroup-root=""
Oct 30 19:16:20.539812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534379    5043 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 19:16:20.539812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534382    5043 flags.go:64] FLAG: --client-ca-file=""
Oct 30 19:16:20.539812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534385    5043 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 19:16:20.539812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534388    5043 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 19:16:20.540402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534395    5043 flags.go:64] FLAG: --cluster-domain=""
Oct 30 19:16:20.540402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534398    5043 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 19:16:20.540402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534402    5043 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 19:16:20.540402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534406    5043 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 19:16:20.540402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534409    5043 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 19:16:20.540402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534413    5043 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 19:16:20.540402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534423    5043 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 19:16:20.540402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534427    5043 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 19:16:20.540402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534430    5043 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 19:16:20.540402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534434    5043 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 19:16:20.540402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534437    5043 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 19:16:20.540402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534440    5043 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 19:16:20.540402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534443    5043 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 19:16:20.540402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534445    5043 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 19:16:20.540402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534450    5043 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 19:16:20.540402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534453    5043 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 19:16:20.540402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534457    5043 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 19:16:20.540402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534460    5043 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 19:16:20.540402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534463    5043 flags.go:64] FLAG: --enable-server="true"
Oct 30 19:16:20.540402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534466    5043 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 19:16:20.540402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534473    5043 flags.go:64] FLAG: --event-burst="100"
Oct 30 19:16:20.540402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534476    5043 flags.go:64] FLAG: --event-qps="50"
Oct 30 19:16:20.540402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534479    5043 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 19:16:20.540402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534482    5043 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 19:16:20.541417 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534485    5043 flags.go:64] FLAG: --eviction-hard=""
Oct 30 19:16:20.541417 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534489    5043 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 19:16:20.541417 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534492    5043 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 19:16:20.541417 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534495    5043 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 19:16:20.541417 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534498    5043 flags.go:64] FLAG: --eviction-soft=""
Oct 30 19:16:20.541417 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534501    5043 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 19:16:20.541417 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534504    5043 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 19:16:20.541417 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534506    5043 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 19:16:20.541417 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534509    5043 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 19:16:20.541417 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534512    5043 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 19:16:20.541417 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534516    5043 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 19:16:20.541417 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534519    5043 flags.go:64] FLAG: --feature-gates=""
Oct 30 19:16:20.541417 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534523    5043 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 19:16:20.541417 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534526    5043 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 19:16:20.541417 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534529    5043 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 19:16:20.541417 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534538    5043 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 19:16:20.541417 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534541    5043 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 19:16:20.541417 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534545    5043 flags.go:64] FLAG: --help="false"
Oct 30 19:16:20.541417 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534548    5043 flags.go:64] FLAG: --hostname-override="ip-10-0-137-84.us-east-2.compute.internal"
Oct 30 19:16:20.541417 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534551    5043 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 19:16:20.541417 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534554    5043 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 19:16:20.541417 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534557    5043 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 19:16:20.541417 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534560    5043 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 19:16:20.542085 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534563    5043 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 19:16:20.542085 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534568    5043 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 19:16:20.542085 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534571    5043 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 19:16:20.542085 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534574    5043 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 19:16:20.542085 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534577    5043 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 19:16:20.542085 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534579    5043 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 19:16:20.542085 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534582    5043 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 19:16:20.542085 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534585    5043 flags.go:64] FLAG: --kube-reserved=""
Oct 30 19:16:20.542085 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534588    5043 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 19:16:20.542085 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534591    5043 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 19:16:20.542085 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534594    5043 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 19:16:20.542085 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534597    5043 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 19:16:20.542085 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534600    5043 flags.go:64] FLAG: --lock-file=""
Oct 30 19:16:20.542085 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534602    5043 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 19:16:20.542085 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534605    5043 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 19:16:20.542085 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534608    5043 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 19:16:20.542085 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534613    5043 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 19:16:20.542085 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534616    5043 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 19:16:20.542085 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534619    5043 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 19:16:20.542085 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534623    5043 flags.go:64] FLAG: --logging-format="text"
Oct 30 19:16:20.542085 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534626    5043 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 19:16:20.542085 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534629    5043 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 19:16:20.542085 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534632    5043 flags.go:64] FLAG: --manifest-url=""
Oct 30 19:16:20.542085 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534635    5043 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 19:16:20.542085 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534639    5043 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 19:16:20.542674 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534648    5043 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 19:16:20.542674 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534652    5043 flags.go:64] FLAG: --max-pods="110"
Oct 30 19:16:20.542674 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534655    5043 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 19:16:20.542674 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534657    5043 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 19:16:20.542674 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534660    5043 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 19:16:20.542674 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534663    5043 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 19:16:20.542674 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534666    5043 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 19:16:20.542674 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534669    5043 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 19:16:20.542674 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534673    5043 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 19:16:20.542674 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534680    5043 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 19:16:20.542674 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534683    5043 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 19:16:20.542674 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534685    5043 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 19:16:20.542674 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534688    5043 flags.go:64] FLAG: --pod-cidr=""
Oct 30 19:16:20.542674 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534691    5043 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 19:16:20.542674 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534694    5043 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 19:16:20.542674 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534697    5043 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 19:16:20.542674 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534699    5043 flags.go:64] FLAG: --port="10250"
Oct 30 19:16:20.542674 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534702    5043 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 19:16:20.542674 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534705    5043 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-02388e4b8a9972580"
Oct 30 19:16:20.542674 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534708    5043 flags.go:64] FLAG: --qos-reserved=""
Oct 30 19:16:20.542674 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534711    5043 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 19:16:20.542674 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534714    5043 flags.go:64] FLAG: --register-node="true"
Oct 30 19:16:20.542674 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534717    5043 flags.go:64] FLAG: --register-with-taints=""
Oct 30 19:16:20.542674 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534721    5043 flags.go:64] FLAG: --registry-burst="10"
Oct 30 19:16:20.542674 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534723    5043 flags.go:64] FLAG: --registry-qps="5"
Oct 30 19:16:20.542674 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534726    5043 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 19:16:20.543314 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534748    5043 flags.go:64] FLAG: --reserved-memory=""
Oct 30 19:16:20.543314 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534752    5043 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 19:16:20.543314 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534755    5043 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 19:16:20.543314 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534758    5043 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 19:16:20.543314 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534761    5043 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 19:16:20.543314 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534764    5043 flags.go:64] FLAG: --runonce="false"
Oct 30 19:16:20.543314 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534767    5043 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 19:16:20.543314 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534776    5043 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 19:16:20.543314 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534779    5043 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 19:16:20.543314 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534783    5043 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 19:16:20.543314 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534785    5043 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 19:16:20.543314 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534789    5043 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 19:16:20.543314 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534792    5043 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 19:16:20.543314 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534794    5043 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 19:16:20.543314 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534798    5043 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 19:16:20.543314 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534801    5043 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 19:16:20.543314 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534804    5043 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 19:16:20.543314 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534807    5043 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 19:16:20.543314 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534810    5043 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 19:16:20.543314 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534813    5043 flags.go:64] FLAG: --system-cgroups=""
Oct 30 19:16:20.543314 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534816    5043 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 19:16:20.543314 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534822    5043 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 19:16:20.543314 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534824    5043 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 19:16:20.543314 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534827    5043 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 19:16:20.543314 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534832    5043 flags.go:64] FLAG: --tls-min-version=""
Oct 30 19:16:20.543977 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534835    5043 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 19:16:20.543977 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534838    5043 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 19:16:20.543977 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534840    5043 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 19:16:20.543977 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534843    5043 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 19:16:20.543977 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534846    5043 flags.go:64] FLAG: --v="2"
Oct 30 19:16:20.543977 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534850    5043 flags.go:64] FLAG: --version="false"
Oct 30 19:16:20.543977 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534855    5043 flags.go:64] FLAG: --vmodule=""
Oct 30 19:16:20.543977 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534860    5043 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 19:16:20.543977 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.534863    5043 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 19:16:20.543977 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535004    5043 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:16:20.543977 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535008    5043 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:16:20.543977 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535011    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:16:20.543977 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535013    5043 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:16:20.543977 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535016    5043 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:16:20.543977 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535018    5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:16:20.543977 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535021    5043 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:16:20.543977 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535025    5043 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:16:20.543977 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535028    5043 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:16:20.543977 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535030    5043 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:16:20.543977 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535034    5043 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:16:20.543977 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535036    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:16:20.543977 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535040    5043 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:16:20.544498 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535043    5043 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:16:20.544498 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535045    5043 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:16:20.544498 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535048    5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:16:20.544498 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535050    5043 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:16:20.544498 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535053    5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:16:20.544498 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535055    5043 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:16:20.544498 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535058    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:16:20.544498 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535060    5043 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:16:20.544498 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535063    5043 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:16:20.544498 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535065    5043 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:16:20.544498 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535068    5043 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:16:20.544498 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535070    5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:16:20.544498 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535073    5043 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:16:20.544498 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535075    5043 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:16:20.544498 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535077    5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:16:20.544498 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535080    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:16:20.544498 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535082    5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:16:20.544498 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535086    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:16:20.544498 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535088    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:16:20.544970 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535090    5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:16:20.544970 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535093    5043 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:16:20.544970 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535095    5043 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:16:20.544970 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535097    5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:16:20.544970 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535100    5043 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:16:20.544970 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535102    5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:16:20.544970 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535104    5043 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:16:20.544970 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535108    5043 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:16:20.544970 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535112    5043 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:16:20.544970 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535115    5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:16:20.544970 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535117    5043 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:16:20.544970 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535120    5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:16:20.544970 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535123    5043 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:16:20.544970 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535126    5043 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:16:20.544970 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535128    5043 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:16:20.544970 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535131    5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:16:20.544970 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535133    5043 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:16:20.544970 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535136    5043 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:16:20.544970 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535138    5043 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:16:20.545432 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535141    5043 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:16:20.545432 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535143    5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:16:20.545432 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535146    5043 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:16:20.545432 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535148    5043 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:16:20.545432 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535151    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:16:20.545432 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535153    5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:16:20.545432 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535156    5043 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:16:20.545432 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535158    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:16:20.545432 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535160    5043 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:16:20.545432 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535163    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:16:20.545432 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535166    5043 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:16:20.545432 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535168    5043 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:16:20.545432 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535170    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:16:20.545432 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535173    5043 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:16:20.545432 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535175    5043 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:16:20.545432 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535178    5043 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:16:20.545432 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535180    5043 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:16:20.545432 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535182    5043 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:16:20.545432 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535185    5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:16:20.545432 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535187    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:16:20.545922 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535190    5043 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:16:20.545922 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535192    5043 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:16:20.545922 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535195    5043 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:16:20.545922 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535197    5043 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:16:20.545922 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535199    5043 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:16:20.545922 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535203    5043 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:16:20.545922 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535205    5043 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:16:20.545922 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535208    5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:16:20.545922 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535210    5043 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:16:20.545922 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535212    5043 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:16:20.545922 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535215    5043 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:16:20.545922 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535217    5043 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:16:20.545922 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535220    5043 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:16:20.545922 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535222    5043 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:16:20.545922 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535225    5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:16:20.545922 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535227    5043 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:16:20.545922 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535230    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:16:20.545922 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535232    5043 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:16:20.545922 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535234    5043 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:16:20.545922 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535237    5043 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:16:20.546393 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535239    5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:16:20.546393 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535241    5043 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:16:20.546393 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535244    5043 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:16:20.546393 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535246    5043 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:16:20.546393 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535248    5043 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:16:20.546393 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535251    5043 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:16:20.546393 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535253    5043 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:16:20.546393 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535256    5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:16:20.546393 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535258    5043 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:16:20.546393 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535260    5043 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:16:20.546393 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535263    5043 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:16:20.546393 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535265    5043 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:16:20.546393 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535268    5043 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:16:20.546393 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535271    5043 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:16:20.546393 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535276    5043 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:16:20.546771 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.535303    5043 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:16:20.546771 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535338    5043 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:16:20.546771 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535346    5043 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:16:20.546771 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535350    5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:16:20.546771 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535353    5043 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:16:20.546771 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535356    5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:16:20.546771 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535358    5043 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:16:20.546771 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535361    5043 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:16:20.546771 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535363    5043 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:16:20.546771 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535366    5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:16:20.546771 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535368    5043 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:16:20.546771 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535371    5043 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:16:20.546771 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535373    5043 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:16:20.546771 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535376    5043 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:16:20.546771 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535378    5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:16:20.546771 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535380    5043 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:16:20.547198 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535383    5043 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:16:20.547198 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535385    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:16:20.547198 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535388    5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:16:20.547198 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535390    5043 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:16:20.547198 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535393    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:16:20.547198 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535395    5043 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:16:20.547198 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535398    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:16:20.547198 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535400    5043 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:16:20.547198 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535403    5043 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:16:20.547198 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535405    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:16:20.547198 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535408    5043 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:16:20.547198 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535410    5043 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:16:20.547198 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535413    5043 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:16:20.547198 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535415    5043 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:16:20.547198 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535418    5043 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:16:20.547198 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535420    5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:16:20.547198 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535423    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:16:20.547198 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535425    5043 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:16:20.547198 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535429    5043 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:16:20.547198 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535431    5043 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:16:20.547757 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535433    5043 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:16:20.547757 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535436    5043 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:16:20.547757 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535438    5043 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:16:20.547757 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535440    5043 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:16:20.547757 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535443    5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:16:20.547757 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535446    5043 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:16:20.547757 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535449    5043 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:16:20.547757 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535451    5043 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:16:20.547757 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535454    5043 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:16:20.547757 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535456    5043 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:16:20.547757 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535459    5043 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:16:20.547757 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535461    5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:16:20.547757 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535464    5043 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:16:20.547757 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535466    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:16:20.547757 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535469    5043 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:16:20.547757 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535471    5043 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:16:20.547757 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535473    5043 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:16:20.547757 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535476    5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:16:20.547757 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535479    5043 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:16:20.547757 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535481    5043 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:16:20.548249 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535484    5043 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:16:20.548249 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535486    5043 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:16:20.548249 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535489    5043 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:16:20.548249 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535491    5043 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:16:20.548249 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535493    5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:16:20.548249 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535496    5043 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:16:20.548249 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535498    5043 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:16:20.548249 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535500    5043 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:16:20.548249 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535503    5043 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:16:20.548249 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535506    5043 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:16:20.548249 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535510    5043 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:16:20.548249 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535514    5043 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:16:20.548249 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535517    5043 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:16:20.548249 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535520    5043 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:16:20.548249 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535523    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:16:20.548249 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535525    5043 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:16:20.548249 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535528    5043 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:16:20.548249 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535531    5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:16:20.548249 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535534    5043 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:16:20.548696 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535537    5043 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:16:20.548696 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535539    5043 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:16:20.548696 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535542    5043 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:16:20.548696 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535544    5043 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:16:20.548696 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535547    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:16:20.548696 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535549    5043 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:16:20.548696 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535551    5043 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:16:20.548696 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535554    5043 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:16:20.548696 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535557    5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:16:20.548696 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535559    5043 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:16:20.548696 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535562    5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:16:20.548696 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535564    5043 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:16:20.548696 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535566    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:16:20.548696 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535569    5043 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:16:20.548696 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535571    5043 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:16:20.548696 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535574    5043 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:16:20.548696 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535576    5043 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:16:20.548696 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535578    5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:16:20.548696 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535581    5043 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:16:20.548696 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535583    5043 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:16:20.549363 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535586    5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:16:20.549363 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535588    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:16:20.549363 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535590    5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:16:20.549363 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535595    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:16:20.549363 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535598    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:16:20.549363 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535600    5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:16:20.549363 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535602    5043 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:16:20.549363 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535605    5043 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:16:20.549363 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535607    5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:16:20.549363 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535610    5043 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:16:20.549363 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535613    5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:16:20.549363 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.535616    5043 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:16:20.549363 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.539474    5043 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 19:16:20.549363 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.539489    5043 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 19:16:20.549363 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.539504    5043 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 19:16:20.549363 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.539508    5043 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 19:16:20.549363 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539565    5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:16:20.549363 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539569    5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:16:20.549363 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539573    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:16:20.549819 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539576    5043 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:16:20.549819 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539578    5043 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:16:20.549819 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539581    5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:16:20.549819 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539583    5043 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:16:20.549819 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539586    5043 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:16:20.549819 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539589    5043 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:16:20.549819 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539591    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:16:20.549819 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539594    5043 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:16:20.549819 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539596    5043 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:16:20.549819 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539599    5043 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:16:20.549819 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539602    5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:16:20.549819 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539604    5043 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:16:20.549819 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539607    5043 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:16:20.549819 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539609    5043 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:16:20.549819 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539612    5043 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:16:20.549819 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539614    5043 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:16:20.549819 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539619    5043 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:16:20.549819 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539623    5043 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:16:20.549819 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539627    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:16:20.550279 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539630    5043 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:16:20.550279 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539634    5043 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:16:20.550279 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539637    5043 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:16:20.550279 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539640    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:16:20.550279 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539644    5043 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:16:20.550279 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539646    5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:16:20.550279 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539650    5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:16:20.550279 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539652    5043 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:16:20.550279 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539655    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:16:20.550279 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539657    5043 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:16:20.550279 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539660    5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:16:20.550279 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539663    5043 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:16:20.550279 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539665    5043 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:16:20.550279 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539667    5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:16:20.550279 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539670    5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:16:20.550279 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539672    5043 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:16:20.550279 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539675    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:16:20.550279 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539678    5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:16:20.550279 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539680    5043 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:16:20.550731 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539682    5043 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:16:20.550731 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539685    5043 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:16:20.550731 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539687    5043 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:16:20.550731 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539690    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:16:20.550731 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539692    5043 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:16:20.550731 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539694    5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:16:20.550731 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539697    5043 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:16:20.550731 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539699    5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:16:20.550731 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539702    5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:16:20.550731 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539704    5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:16:20.550731 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539707    5043 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:16:20.550731 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539709    5043 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:16:20.550731 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539712    5043 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:16:20.550731 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539715    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:16:20.550731 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539717    5043 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:16:20.550731 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539720    5043 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:16:20.550731 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539722    5043 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:16:20.550731 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539724    5043 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:16:20.550731 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539727    5043 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:16:20.551195 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539729    5043 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:16:20.551195 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539754    5043 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:16:20.551195 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539759    5043 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:16:20.551195 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539761    5043 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:16:20.551195 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539764    5043 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:16:20.551195 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539766    5043 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:16:20.551195 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539769    5043 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:16:20.551195 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539771    5043 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:16:20.551195 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539774    5043 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:16:20.551195 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539776    5043 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:16:20.551195 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539779    5043 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:16:20.551195 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539781    5043 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:16:20.551195 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539784    5043 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:16:20.551195 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539786    5043 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:16:20.551195 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539789    5043 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:16:20.551195 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539791    5043 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:16:20.551195 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539794    5043 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:16:20.551195 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539796    5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:16:20.551195 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539799    5043 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:16:20.551195 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539801    5043 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:16:20.551669 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539804    5043 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:16:20.551669 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539806    5043 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:16:20.551669 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539809    5043 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:16:20.551669 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539812    5043 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:16:20.551669 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539814    5043 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:16:20.551669 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539818    5043 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:16:20.551669 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539821    5043 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:16:20.551669 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539823    5043 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:16:20.551669 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539826    5043 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:16:20.551669 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539829    5043 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:16:20.551669 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539831    5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:16:20.551669 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539834    5043 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:16:20.551669 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539836    5043 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:16:20.551669 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539838    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:16:20.551669 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539842    5043 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:16:20.551669 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539845    5043 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:16:20.551669 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539848    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:16:20.551669 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539850    5043 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:16:20.551669 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539853    5043 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:16:20.551669 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539855    5043 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:16:20.552232 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539868    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:16:20.552232 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539871    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:16:20.552232 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539873    5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:16:20.552232 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539876    5043 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:16:20.552232 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539879    5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:16:20.552232 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.539881    5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:16:20.552232 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.539910    5043 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:16:20.552232 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540048    5043 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:16:20.552232 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540052    5043 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:16:20.552232 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540055    5043 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:16:20.552232 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540057    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:16:20.552232 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540060    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:16:20.552232 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540062    5043 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:16:20.552232 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540064    5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:16:20.552232 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540067    5043 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:16:20.552232 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540069    5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:16:20.552612 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540072    5043 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:16:20.552612 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540074    5043 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:16:20.552612 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540077    5043 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:16:20.552612 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540080    5043 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:16:20.552612 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540082    5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:16:20.552612 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540085    5043 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:16:20.552612 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540087    5043 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:16:20.552612 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540089    5043 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:16:20.552612 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540095    5043 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:16:20.552612 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540097    5043 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:16:20.552612 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540100    5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:16:20.552612 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540102    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:16:20.552612 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540105    5043 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:16:20.552612 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540108    5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:16:20.552612 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540110    5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:16:20.552612 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540113    5043 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:16:20.552612 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540116    5043 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:16:20.552612 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540118    5043 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:16:20.552612 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540120    5043 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:16:20.552612 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540123    5043 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:16:20.553185 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540125    5043 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:16:20.553185 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540128    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:16:20.553185 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540130    5043 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:16:20.553185 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540133    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:16:20.553185 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540135    5043 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:16:20.553185 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540138    5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:16:20.553185 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540140    5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:16:20.553185 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540143    5043 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:16:20.553185 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540145    5043 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:16:20.553185 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540147    5043 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:16:20.553185 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540150    5043 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:16:20.553185 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540152    5043 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:16:20.553185 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540155    5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:16:20.553185 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540157    5043 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:16:20.553185 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540159    5043 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:16:20.553185 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540163    5043 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:16:20.553185 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540165    5043 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:16:20.553185 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540168    5043 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:16:20.553185 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540170    5043 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:16:20.553185 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540173    5043 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:16:20.553669 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540178    5043 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:16:20.553669 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540180    5043 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:16:20.553669 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540183    5043 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:16:20.553669 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540185    5043 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:16:20.553669 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540188    5043 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:16:20.553669 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540190    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:16:20.553669 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540193    5043 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:16:20.553669 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540195    5043 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:16:20.553669 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540198    5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:16:20.553669 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540200    5043 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:16:20.553669 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540203    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:16:20.553669 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540205    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:16:20.553669 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540208    5043 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:16:20.553669 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540210    5043 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:16:20.553669 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540213    5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:16:20.553669 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540215    5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:16:20.553669 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540218    5043 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:16:20.553669 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540220    5043 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:16:20.553669 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540223    5043 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:16:20.554172 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540225    5043 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:16:20.554172 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540228    5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:16:20.554172 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540230    5043 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:16:20.554172 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540233    5043 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:16:20.554172 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540235    5043 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:16:20.554172 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540238    5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:16:20.554172 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540240    5043 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:16:20.554172 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540244    5043 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:16:20.554172 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540247    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:16:20.554172 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540251    5043 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:16:20.554172 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540253    5043 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:16:20.554172 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540256    5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:16:20.554172 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540258    5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:16:20.554172 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540262    5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:16:20.554172 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540265    5043 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:16:20.554172 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540268    5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:16:20.554172 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540270    5043 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:16:20.554172 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540273    5043 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:16:20.554172 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540275    5043 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:16:20.554622 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540277    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:16:20.554622 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540280    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:16:20.554622 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540282    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:16:20.554622 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540285    5043 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:16:20.554622 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540287    5043 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:16:20.554622 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540290    5043 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:16:20.554622 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540292    5043 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:16:20.554622 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540295    5043 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:16:20.554622 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540297    5043 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:16:20.554622 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540300    5043 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:16:20.554622 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540302    5043 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:16:20.554622 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540305    5043 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:16:20.554622 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540308    5043 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:16:20.554622 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540311    5043 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:16:20.554622 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540315    5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:16:20.554622 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540319    5043 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:16:20.554622 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540321    5043 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:16:20.554622 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540324    5043 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:16:20.554622 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540326    5043 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:16:20.555080 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.540349    5043 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:16:20.555080 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540765    5043 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:16:20.555080 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540780    5043 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:16:20.555080 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540786    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:16:20.555080 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540810    5043 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:16:20.555080 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540814    5043 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:16:20.555080 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540818    5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:16:20.555080 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540823    5043 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:16:20.555080 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540830    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:16:20.555080 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540834    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:16:20.555080 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540838    5043 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:16:20.555080 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540843    5043 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:16:20.555080 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540847    5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:16:20.555080 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540851    5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:16:20.555080 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540855    5043 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:16:20.555080 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540860    5043 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:16:20.555453 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540870    5043 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:16:20.555453 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540874    5043 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:16:20.555453 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540878    5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:16:20.555453 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540883    5043 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:16:20.555453 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540889    5043 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:16:20.555453 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540893    5043 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:16:20.555453 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540898    5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:16:20.555453 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540902    5043 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:16:20.555453 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540909    5043 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:16:20.555453 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540915    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:16:20.555453 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540920    5043 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:16:20.555453 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540924    5043 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:16:20.555453 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540929    5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:16:20.555453 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540937    5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:16:20.555453 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540942    5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:16:20.555453 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540946    5043 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:16:20.555453 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540950    5043 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:16:20.555453 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540955    5043 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:16:20.555453 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540959    5043 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:16:20.556032 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540963    5043 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:16:20.556032 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540967    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:16:20.556032 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540973    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:16:20.556032 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540977    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:16:20.556032 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540981    5043 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:16:20.556032 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540989    5043 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:16:20.556032 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.540998    5043 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:16:20.556032 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541003    5043 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:16:20.556032 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541007    5043 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:16:20.556032 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541011    5043 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:16:20.556032 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541016    5043 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:16:20.556032 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541020    5043 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:16:20.556032 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541024    5043 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:16:20.556032 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541029    5043 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:16:20.556032 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541035    5043 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:16:20.556032 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541041    5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:16:20.556032 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541046    5043 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:16:20.556032 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541053    5043 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:16:20.556032 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541062    5043 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:16:20.556485 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541067    5043 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:16:20.556485 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541071    5043 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:16:20.556485 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541075    5043 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:16:20.556485 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541080    5043 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:16:20.556485 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541084    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:16:20.556485 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541088    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:16:20.556485 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541092    5043 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:16:20.556485 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541096    5043 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:16:20.556485 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541100    5043 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:16:20.556485 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541104    5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:16:20.556485 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541109    5043 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:16:20.556485 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541113    5043 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:16:20.556485 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541121    5043 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:16:20.556485 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541126    5043 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:16:20.556485 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541130    5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:16:20.556485 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541134    5043 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:16:20.556485 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541144    5043 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:16:20.556485 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541148    5043 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:16:20.556485 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541154    5043 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:16:20.556485 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541158    5043 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:16:20.556991 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541162    5043 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:16:20.556991 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541167    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:16:20.556991 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541171    5043 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:16:20.556991 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541176    5043 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:16:20.556991 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541180    5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:16:20.556991 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541190    5043 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:16:20.556991 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541194    5043 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:16:20.556991 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541198    5043 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:16:20.556991 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541203    5043 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:16:20.556991 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541207    5043 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:16:20.556991 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541213    5043 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:16:20.556991 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541218    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:16:20.556991 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541222    5043 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:16:20.556991 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541226    5043 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:16:20.556991 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541231    5043 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:16:20.556991 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541235    5043 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:16:20.556991 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541240    5043 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:16:20.556991 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541252    5043 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:16:20.556991 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541256    5043 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:16:20.556991 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541261    5043 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:16:20.557466 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541265    5043 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:16:20.557466 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541269    5043 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:16:20.557466 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541274    5043 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:16:20.557466 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541278    5043 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:16:20.557466 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541282    5043 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:16:20.557466 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541286    5043 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:16:20.557466 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541290    5043 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:16:20.557466 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541295    5043 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:16:20.557466 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541299    5043 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:16:20.557466 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541309    5043 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:16:20.557466 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541314    5043 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:16:20.557466 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541318    5043 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:16:20.557466 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:20.541332    5043 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:16:20.557466 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.541456    5043 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 19:16:20.557466 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.542845    5043 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 19:16:20.557466 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.542902    5043 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 19:16:20.557466 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.543095    5043 server.go:1014] "Starting client certificate rotation"
Oct 30 19:16:20.557466 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.543229    5043 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 19:16:20.557960 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.543278    5043 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:10:38 +0000 UTC" deadline="2028-04-28 11:14:43.791493986 +0000 UTC"
Oct 30 19:16:20.557960 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.543305    5043 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13095h58m23.2481902s"
Oct 30 19:16:20.557960 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.543560    5043 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:16:20.569433 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.569407    5043 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:16:20.570721 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.570706    5043 log.go:25] "Validated CRI v1 runtime API"
Oct 30 19:16:20.572467 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.572452    5043 log.go:25] "Validated CRI v1 image API"
Oct 30 19:16:20.573396 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.573370    5043 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 19:16:20.575421 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.575396    5043 fs.go:135] Filesystem UUIDs: map[543baacd-93ee-48e6-a462-351dd9dea122:/dev/nvme0n1p4 7286354b-3eef-4091-a771-eee36e8f6f81:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Oct 30 19:16:20.575668 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.575423    5043 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/012f367ad67d33090ad1abbbf367716fe83ab03857802f0c325e08d1b002f679/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/012f367ad67d33090ad1abbbf367716fe83ab03857802f0c325e08d1b002f679/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/087138c34823747582e3393ac259cb9bb6078aad337a07f0de1831b5b7b1c0bf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/087138c34823747582e3393ac259cb9bb6078aad337a07f0de1831b5b7b1c0bf/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/103e94309533e5b3f462ba5781ecee4e8885c5ccf33c62bf1f560d1f596b95df/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/103e94309533e5b3f462ba5781ecee4e8885c5ccf33c62bf1f560d1f596b95df/userdata/shm major:0 minor:345 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/14b01e2ba44a03c9687bf7ae9ad7d318310a5472f8b4a12c835a6894a525714a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/14b01e2ba44a03c9687bf7ae9ad7d318310a5472f8b4a12c835a6894a525714a/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/16a270602f3edfacb90114f8ac0b1d3a49a4c3dd5d021f56175b526b2136d7e2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/16a270602f3edfacb90114f8ac0b1d3a49a4c3dd5d021f56175b526b2136d7e2/userdata/shm major:0 minor:79 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1b81426a4a1dfc6148525fe704c7fc993cfeb9bd6a25c91c4ba435fbafe15bce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1b81426a4a1dfc6148525fe704c7fc993cfeb9bd6a25c91c4ba435fbafe15bce/userdata/shm major:0 minor:342 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2268a2c217d2bf4d82f1c06fefc397fff88c6c383fddbdaa5c240ca4a9fdde56/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2268a2c217d2bf4d82f1c06fefc397fff88c6c383fddbdaa5c240ca4a9fdde56/userdata/shm major:0 minor:356 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3367f10f4ffc027c5f101ffb7619c5ebe46b3e6ea784512fb62472c7e9c9fd71/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3367f10f4ffc027c5f101ffb7619c5ebe46b3e6ea784512fb62472c7e9c9fd71/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/487fa79e6ceaed2223a66678071091aae7f373b853288eb4e2de23616b76a96b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/487fa79e6ceaed2223a66678071091aae7f373b853288eb4e2de23616b76a96b/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5b80a342a983dcc8d8c93e5a7498eb7deed13dd3ae040b10a76c10a2c4c52644/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5b80a342a983dcc8d8c93e5a7498eb7deed13dd3ae040b10a76c10a2c4c52644/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/751934796f1ae1679278572b9ef2bb6bb010c50c66c514be4b59fa4e90ca381c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/751934796f1ae1679278572b9ef2bb6bb010c50c66c514be4b59fa4e90ca381c/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7cf459bbec6deed978519f5d53895993cfcc83c8692fe905e753eb4122488bd9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7cf459bbec6deed978519f5d53895993cfcc83c8692fe905e753eb4122488bd9/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/857baf1dd5d42d5472787a331e87333fc9cf66d405389472085f2803f7a4af2c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/857baf1dd5d42d5472787a331e87333fc9cf66d405389472085f2803f7a4af2c/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ad4d9072a8cb0ef7d9197f2a1afef0c121f90b30eefd4f2ec780bfec7a063d4c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ad4d9072a8cb0ef7d9197f2a1afef0c121f90b30eefd4f2ec780bfec7a063d4c/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b4cf0b0eda03520a427f16b5b09773069d86edc6466c727314bbc46f8bdfb62f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b4cf0b0eda03520a427f16b5b09773069d86edc6466c727314bbc46f8bdfb62f/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bbe001d2613d79b8c9b2749f057a8249b324e97992746982fb7261e20a45458e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bbe001d2613d79b8c9b2749f057a8249b324e97992746982fb7261e20a45458e/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c7612191c1ecc48d5216a96ecbbc00bea70bdd93f79957dcfef867e1f359dbf9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c7612191c1ecc48d5216a96ecbbc00bea70bdd93f79957dcfef867e1f359dbf9/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c7a27c1c553a6b4378a8b33e73553bea17b222b6c46e2fdf059823d42abb7f41/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c7a27c1c553a6b4378a8b33e73553bea17b222b6c46e2fdf059823d42abb7f41/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c8d548b04e3d8e8c629730de6fc83598257db1ec337aabbb0514d1d0d007a668/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c8d548b04e3d8e8c629730de6fc83598257db1ec337aabbb0514d1d0d007a668/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c8faece2f3197dd85421545da5f23e9e2d01747c378c197afd296e70c026d12a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c8faece2f3197dd85421545da5f23e9e2d01747c378c197afd296e70c026d12a/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ca59d13272efc207f8bd73365bf08e0afe178dc4ec377b5ff544d3d9468374a7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ca59d13272efc207f8bd73365bf08e0afe178dc4ec377b5ff544d3d9468374a7/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d4e2c0ea2c6d38835a0854694a88a86923bd0ddbf7884508c97c088b14da698a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d4e2c0ea2c6d38835a0854694a88a86923bd0ddbf7884508c97c088b14da698a/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eab4e3466e8405b2944cdfca50c0dc8782020e6afc0d3ec51310419af686226e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eab4e3466e8405b2944cdfca50c0dc8782020e6afc0d3ec51310419af686226e/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fb1b480febdfaa3569526b9c3c98e490125407a1721d4f174e0eb8d77d707b74/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fb1b480febdfaa3569526b9c3c98e490125407a1721d4f174e0eb8d77d707b74/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fe12e0dde11c16433babd16be15e78a1bce1c00b6854cf7addaa3503d314dd5a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fe12e0dde11c16433babd16be15e78a1bce1c00b6854cf7addaa3503d314dd5a/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ff2ebe84c24d4fd9301608b39c1fd681d19907a9cdea1f6c818f44762d03b0a9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ff2ebe84c24d4fd9301608b39c1fd681d19907a9cdea1f6c818f44762d03b0a9/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/00b97df0-857e-4ad9-b33a-4a59777b6f62/volumes/kubernetes.io~projected/kube-api-access-w8ww9:{mountpoint:/var/lib/kubelet/pods/00b97df0-857e-4ad9-b33a-4a59777b6f62/volumes/kubernetes.io~projected/kube-api-access-w8ww9 major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0269ffa6-ce06-43f3-be2a-58f6aa3e6964/volumes/kubernetes.io~projected/kube-api-access-b6bsd:{mountpoint:/var/lib/kubelet/pods/0269ffa6-ce06-43f3-be2a-58f6aa3e6964/volumes/kubernetes.io~projected/kube-api-access-b6bsd major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/06397fd4-3377-4eaa-8879-ebbb5943fbda/volumes/kubernetes.io~projected/kube-api-access-zh979:{mountpoint:/var/lib/kubelet/pods/06397fd4-3377-4eaa-8879-ebbb5943fbda/volumes/kubernetes.io~projected/kube-api-access-zh979 major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/128a8dad-66f8-414b-8c1d-14c7d3b58d25/volumes/kubernetes.io~projected/kube-api-access-jxkzk:{mountpoint:/var/lib/kubelet/pods/128a8dad-66f8-414b-8c1d-14c7d3b58d25/volumes/kubernetes.io~projected/kube-api-access-jxkzk major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16705cc4-1ecc-47d1-9f4a-bb7eadc6294c/volumes/kubernetes.io~projected/kube-api-access-qr447:{mountpoint:/var/lib/kubelet/pods/16705cc4-1ecc-47d1-9f4a-bb7eadc6294c/volumes/kubernetes.io~projected/kube-api-access-qr447 major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/247ff425-bed3-4297-b5c0-cc32b21bc179/volumes/kubernetes.io~projected/kube-api-access-28sgd:{mountpoint:/var/lib/kubelet/pods/247ff425-bed3-4297-b5c0-cc32b21bc179/volumes/kubernetes.io~projected/kube-api-access-28sgd major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/247ff425-bed3-4297-b5c0-cc32b21bc179/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/247ff425-bed3-4297-b5c0-cc32b21bc179/volumes/kubernetes.io~secret/default-certificate major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/247ff425-bed3-4297-b5c0-cc32b21bc179/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/247ff425-bed3-4297-b5c0-cc32b21bc179/volumes/kubernetes.io~secret/stats-auth major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d261662-727d-4d68-9f4a-9790ecf16d46/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/2d261662-727d-4d68-9f4a-9790ecf16d46/volumes/kubernetes.io~secret/agent-certs major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d52a94d-72ac-4e5c-9472-104399a21a3b/volumes/kubernetes.io~projected/kube-api-access-zfglw:{mountpoint:/var/lib/kubelet/pods/2d52a94d-72ac-4e5c-9472-104399a21a3b/volumes/kubernetes.io~projected/kube-api-access-zfglw major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/353ebfb4-7e11-41de-b992-277443f56960/volumes/kubernetes.io~projected/kube-api-access-2pfsl:{mountpoint:/var/lib/kubelet/pods/353ebfb4-7e11-41de-b992-277443f56960/volumes/kubernetes.io~projected/kube-api-access-2pfsl major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/353ebfb4-7e11-41de-b992-277443f56960/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/353ebfb4-7e11-41de-b992-277443f56960/volumes/kubernetes.io~secret/serving-cert major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36ab52a2-65e2-4af6-a872-9052e12714d1/volumes/kubernetes.io~projected/kube-api-access-lksw9:{mountpoint:/var/lib/kubelet/pods/36ab52a2-65e2-4af6-a872-9052e12714d1/volumes/kubernetes.io~projected/kube-api-access-lksw9 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47e9832e-bc96-445a-a93e-9d24f84db3b3/volumes/kubernetes.io~projected/kube-api-access-6gnlf:{mountpoint:/var/lib/kubelet/pods/47e9832e-bc96-445a-a93e-9d24f84db3b3/volumes/kubernetes.io~projected/kube-api-access-6gnlf major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4dbf226e-336f-42c6-a766-cc0ddfe68bef/volumes/kubernetes.io~projected/kube-api-access-lbndd:{mountpoint:/var/lib/kubelet/pods/4dbf226e-336f-42c6-a766-cc0ddfe68bef/volumes/kubernetes.io~projected/kube-api-access-lbndd major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5338b461-6b7c-474b-8458-d1b53775ee0f/volumes/kubernetes.io~projected/kube-api-access-4xrrr:{mountpoint:/var/lib/kubelet/pods/5338b461-6b7c-474b-8458-d1b53775ee0f/volumes/kubernetes.io~projected/kube-api-access-4xrrr major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6331f51e-cf5e-4ba2-9eda-35fd62b14091/volumes/kubernetes.io~projected/kube-api-access-q5ldg:{mountpoint:/var/lib/kubelet/pods/6331f51e-cf5e-4ba2-9eda-35fd62b14091/volumes/kubernetes.io~projected/kube-api-access-q5ldg major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68201777-2ec5-43b2-b587-81e585865d08/volumes/kubernetes.io~projected/kube-api-access-s56kf:{mountpoint:/var/lib/kubelet/pods/68201777-2ec5-43b2-b587-81e585865d08/volumes/kubernetes.io~projected/kube-api-access-s56kf major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68201777-2ec5-43b2-b587-81e585865d08/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/68201777-2ec5-43b2-b587-81e585865d08/volumes/kubernetes.io~secret/serving-cert major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ecb643b-1dce-4523-89df-93b22ad8ce6f/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/6ecb643b-1dce-4523-89df-93b22ad8ce6f/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/765d8d9d-b89f-419f-b237-3f8034db1bed/volumes/kubernetes.io~projected/kube-api-access-njpgh:{mountpoint:/var/lib/kubelet/pods/765d8d9d-b89f-419f-b237-3f8034db1bed/volumes/kubernetes.io~projected/kube-api-access-njpgh major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/765d8d9d-b89f-419f-b237-3f8034db1bed/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/765d8d9d-b89f-419f-b237-3f8034db1bed/volumes/kubernetes.io~secret/cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76e21916-0744-4499-a0cb-64a7a44a8285/volumes/kubernetes.io~projected/kube-api-access-v6chk:{mountpoint:/var/lib/kubelet/pods/76e21916-0744-4499-a0cb-64a7a44a8285/volumes/kubernetes.io~projected/kube-api-access-v6chk major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76e21916-0744-4499-a0cb-64a7a44a8285/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/76e21916-0744-4499-a0cb-64a7a44a8285/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/77fbf490-7af3-4bde-8222-4038e2219fff/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/77fbf490-7af3-4bde-8222-4038e2219fff/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/77fbf490-7af3-4bde-8222-4038e2219fff/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/77fbf490-7af3-4bde-8222-4038e2219fff/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/77fbf490-7af3-4bde-8222-4038e2219fff/volumes/kubernetes.io~projected/kube-api-access-l8ptq:{mountpoint:/var/lib/kubelet/pods/77fbf490-7af3-4bde-8222-4038e2219fff/volumes/kubernetes.io~projected/kube-api-access-l8ptq major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/853a64ec-147d-42eb-8946-e335de7878d3/volumes/kubernetes.io~projected/kube-api-access-7hx4s:{mountpoint:/var/lib/kubelet/pods/853a64ec-147d-42eb-8946-e335de7878d3/volumes/kubernetes.io~projected/kube-api-access-7hx4s major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9201f6f2-73b9-45cd-9950-3a07914d8ea9/volumes/kubernetes.io~projected/kube-api-access-ndw2r:{mountpoint:/var/lib/kubelet/pods/9201f6f2-73b9-45cd-9950-3a07914d8ea9/volumes/kubernetes.io~projected/kube-api-access-ndw2r major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f302411-01be-438c-a30c-70c944afd630/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/9f302411-01be-438c-a30c-70c944afd630/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f302411-01be-438c-a30c-70c944afd630/volumes/kubernetes.io~projected/kube-api-access-hpb5t:{mountpoint:/var/lib/kubelet/pods/9f302411-01be-438c-a30c-70c944afd630/volumes/kubernetes.io~projected/kube-api-access-hpb5t major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f302411-01be-438c-a30c-70c944afd630/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/9f302411-01be-438c-a30c-70c944afd630/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f302411-01be-438c-a30c-70c944afd630/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/9f302411-01be-438c-a30c-70c944afd630/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a37bca55-21cf-4b76-9c05-e2c951647bf8/volumes/kubernetes.io~projected/kube-api-access-2vk66:{mountpoint:/var/lib/kubelet/pods/a37bca55-21cf-4b76-9c05-e2c951647bf8/volumes/kubernetes.io~projected/kube-api-access-2vk66 major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a37bca55-21cf-4b76-9c05-e2c951647bf8/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/a37bca55-21cf-4b76-9c05-e2c951647bf8/volumes/kubernetes.io~secret/signing-key major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a64211d0-1c48-41d2-b254-4ae84d5097d7/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/a64211d0-1c48-41d2-b254-4ae84d5097d7/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a64211d0-1c48-41d2-b254-4ae84d5097d7/volumes/kubernetes.io~projected/kube-api-access-6x795:{mountpoint:/var/lib/kubelet/pods/a64211d0-1c48-41d2-b254-4ae84d5097d7/volumes/kubernetes.io~projected/kube-api-access-6x795 major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a64211d0-1c48-41d2-b254-4ae84d5097d7/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/a64211d0-1c48-41d2-b254-4ae84d5097d7/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a64211d0-1c48-41d2-b254-4ae84d5097d7/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/a64211d0-1c48-41d2-b254-4ae84d5097d7/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aeef13dc-e06e-49c5-b893-64c7f73a7e44/volumes/kubernetes.io~projected/kube-api-access-tq4kp:{mountpoint:/var/lib/kubelet/pods/aeef13dc-e06e-49c5-b893-64c7f73a7e44/volumes/kubernetes.io~projected/kube-api-access-tq4kp major:0 minor:321 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c243f1d8-1c43-4eb8-93ec-cb28d72cc8a1/volumes/kubernetes.io~projected/kube-api-access-4tnx8:{mountpoint:/var/lib/kubelet/pods/c243f1d8-1c43-4eb8-93ec-cb28d72cc8a1/volumes/kubernetes.io~projected/kube-api-access-4tnx8 major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c84a5c22-7a46-4de1-aef7-32cb9a5453bb/volumes/kubernetes.io~projected/kube-api-access-zsjcc:{mountpoint:/var/lib/kubelet/pods/c84a5c22-7a46-4de1-aef7-32cb9a5453bb/volumes/kubernetes.io~projected/kube-api-access-zsjcc major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cbc4f7c7-bebd-4200-833e-1afbf3c2c81a/volumes/kubernetes.io~projected/kube-api-access-mcgjp:{mountpoint:/var/lib/kubelet/pods/cbc4f7c7-bebd-4200-833e-1afbf3c2c81a/volumes/kubernetes.io~projected/kube-api-access-mcgjp major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc581227-9542-4571-aced-63d191692a41/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/dc581227-9542-4571-aced-63d191692a41/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc581227-9542-4571-aced-63d191692a41/volumes/kubernetes.io~projected/kube-api-access-4m9lt:{mountpoint:/var/lib/kubelet/pods/dc581227-9542-4571-aced-63d191692a41/volumes/kubernetes.io~projected/kube-api-access-4m9lt major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc581227-9542-4571-aced-63d191692a41/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/dc581227-9542-4571-aced-63d191692a41/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dda4158a-327b-4b73-8673-3df1c1c0e400/volumes/kubernetes.io~projected/kube-api-access-m5c96:{mountpoint:/var/lib/kubelet/pods/dda4158a-327b-4b73-8673-3df1c1c0e400/volumes/kubernetes.io~projected/kube-api-access-m5c96 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5002691-8257-4dab-a88b-3c1b60e8e6f8/volumes/kubernetes.io~projected/kube-api-access-9rz2d:{mountpoint:/var/lib/kubelet/pods/e5002691-8257-4dab-a88b-3c1b60e8e6f8/volumes/kubernetes.io~projected/kube-api-access-9rz2d major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5002691-8257-4dab-a88b-3c1b60e8e6f8/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/e5002691-8257-4dab-a88b-3c1b60e8e6f8/volumes/kubernetes.io~secret/serving-cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8e297a2-a78a-4665-86d0-6d842bbcaa51/volumes/kubernetes.io~projected/kube-api-access-5xfxj:{mountpoint:/var/lib/kubelet/pods/e8e297a2-a78a-4665-86d0-6d842bbcaa51/volumes/kubernetes.io~projected/kube-api-access-5xfxj major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f1ea27b2-80df-404c-ac39-460f495957b2/volumes/kubernetes.io~projected/kube-api-access-tngcd:{mountpoint:/var/lib/kubelet/pods/f1ea27b2-80df-404c-ac39-460f495957b2/volumes/kubernetes.io~projected/kube-api-access-tngcd major:0 minor:65 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/f767bd6c7dfd3079a4e2c9b9328d8ae7c9b4d3bd882deb1548424ba509ad63aa/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/c62d5612ceb017ad72a1d5c1cd0f3bf8ff52305b0e5443b87fca45e2c9d0ca7c/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/b5161ef30ec626b1ee0ecaf16b6d7067528631325db7dfa04ff3c68cfd39a4eb/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/5087d2341ac6910efaa3160bfb62034b4aa3550d0559c6ad79fcf1c04c350b86/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/c3c21e1eb367c919f0b6af43db92331136472d9529abaf85009f4a4d8d0bda8a/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/a1afc647239361e435d6b0945096465553671e06c8f39282cebfc7c0bbdf491a/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/7658d68879d3dd3a8d981554507be55a41c878cef8aafa079b992f50289c5928/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/d80cc1c4aa74b1f04b095e2b3563f525eb82fafd59bbfd9e0100758afa2c64a7/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/125865e5ab4d15480fc3f2ac24ad5ac59a57b114526444899c6bcdedfc0e47ba/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/f3d488c1b1de241e5080d57452d92f7ea8b752228d434d0a3a8dbfece35d0419/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/eafa546e5b892886808e207f8e4073ff4ff9af749123cb8345bcfa3a94a42cf5/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/6a5c7e7fd52e77a1a274354e9c8d780bbca7a016d4a21d839539bf6d96a67918/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/234caa08e7336f980b2d33078293d085ee7892e27d5a1c25e6377a8688d68757/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/6361c715576190abff546f7be183a62b01d40496fbd49b7d54df2a5eae78e0a6/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/0909f96be4d536f1c5ff9976aa9f1e668974133f525ea264325ee45708433f7a/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/f68511013a76b1a58ae145ffbe253c1c87c36bd8fd7f51f7e65717f9f8e5b981/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/a0feea6c3a6d2e728c9a1d704a0fab4974603cc4bc9855ae8f0edbef921a5598/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/c1b68007d2a8fc8d278056e0b9a8f91a2306c15046ca7920d4e42f6d406c4a66/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-181:{mountpoint:/var/lib/containers/storage/overlay/603b3c99bbab3a154546e90521861f8db6bcd754df6b00f57dc19fb8342484df/merged major:0 minor:181 fsType:overlay blockSize:0} overlay_0-184:{mountpoint:/var/lib/containers/storage/overlay/f596fa6f1ba9a67cd045fe0800b994188bb1b47ece53785c34859458323ed588/merged major:0 minor:184 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/ac7f55373f72de18346acfcf4823796cbcceec646b5eb25933619bf1b23ca274/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/4895a2f8b6dc440d570cae5aecef1b718db6efa08b7f23685f67bee5c875455d/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/030bd2593ca832c9449fbb9d5b4ae86c8c209e920d5edbf4915749b4a660e6ba/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/908b7468ac66dec2f9aec76d4ac10d05f5cae8fb4dd3fcbe63c75cee07bc3dca/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/246a340f75ca36a3c50e80293fd3bfda088e77ba0a36197537a96d682c376937/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/6bd6cd81a0809d0ae244d3db9f498daa0ec1c9876e48a8978ab80e225d0b6daf/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/eefc10a202cdced8ecd21ecdaa7091b5201276f85e829787137f9b132e3b699a/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/8fc64b37ece9002fbc0eeac6e39a2cfa577b06674fd4876613f1015b912774af/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/90307daf0aa29ba067ad078850f1ea62026c649825efe299b36316394c9373b1/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/f1aa94084d5343725a90712f3ff728041144bd3260fe2dea71bbf52303e5069c/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/a92794aff1806bd87baa916f5eeddf17cdf1fbf17651ce15be9f738e0dc58eb9/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/c43ced19510a83d159293da08f6d7f2afedec4562e552b261ebaafeb4a436f3f/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/95c897e3cd6d447679073faf41853b98b42ceb2ceedea2ab449f189ad5402676/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/943a8e83726f867d4a588fe4d239436224b2ac78c49fbcee94c3ceea379329d4/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/9c0a9dc6f9c8b2d653353e79cdfe89e83a2932556137c56e404ddf082cd90b3e/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/7cd69f7de4f5651cd907803023c7e3196e6666393c8bbe47e070939c7fd90c4b/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/1c6f613fe124277d27f4310c22927d6b1da909a61a60956b53146aadf4db2138/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/0191e1104e07bf8e3d30534c5f10244b2e8f98399e0d77a77842abc7bec686d3/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/36ebdc26200174a0bb7c9ef50b06e764f781ffc9a070cfe0dbc628b958e0b88c/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/e17bdf8c6c617626037dd1dc39adbddbfb59a8e233ffc665f32734d1ab8f4b6b/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/16bc087a19843792d3f2c78e963b575f72a9ccd8127fd26093a2eba481cfc4d9/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/bf2001ea49ead8a0ee45b53c527265d2bc6c80e5fbc03e0ab00b1faef85fa27e/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/83c5fc8ec759dd63dfd912836e184d27ba1d17f39b5cad270fe854a54616510a/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/017f1584e6d1f4cf95313c10f0dbfa899a75cf817b6b2e48c65246527a1e0b60/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/cb8afa0ef1756c4ef37f8c499a9646f2a52937f8695c18804e7018b39df5396f/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/ef81158b60f9ed8ce17b8a566b4e4079a2bf444fd50948ed3a30c646431b5116/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/8dd21a98d1cf39ddc3925ef11431df17c4c10ad36b2caf49f0927b536c92af53/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/9323fa7161e6643b2ec05314b258c6e846c47ab10972cf375ead7655e2bbc00e/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/53fbd2fd3d03a1d0a77166beb748753dd45a1cd873c0c58d1a4c90873ae30e56/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/0b2cbf633eb60b63160e299b2bd728830d665a69ef508d98b15cc04e94b7d914/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/ef55bc24f38f614e4314b70e3745633a84644853f34c06e22b357d226402da9d/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/3acafd4514432ea8459c7c2c83991c12f02ccd08839384872e4c1ed544824405/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/7311746edf80ae8a7d43352fb4c4d08099ce71dd69539302d1be045614214725/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/aa12625a4a484c3bf6cb1d5da567396bfedb2ae3029dd34f38456bd295cfad8a/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/d7d90cff579be5248791e41fd01c42b46b49b66497647f12087781388ff29de9/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/1f6e1b7d6fc1a6107e01d1d1af2218e4f0250bda56095e337003d2a16f0221f8/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/20c20191c7b715b58ec42b66c56d0da1f67e2603c686609c226129ee31e1451a/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-81:{mountpoint:/var/lib/containers/storage/overlay/a1f048146a745a15a62ff358dde72e532f5f5f5cd76361437053b151236f8330/merged major:0 minor:81 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/22380fe2de83403fb4a267b80728e5387979d9079604eddc4452135f5dafa67d/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/bc30ec1c2a9327af33b47e2666b2daa8ab3041fc3737713d9c932ebeb5caef8e/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/46b1d31b04855de65ddc865749208253cc29f0d0162ed323007993a551e8f33d/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/8f952d11c117e9e406acbf147ed4e3691be2d125666bce5ff9c748e765645044/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 19:16:20.584304 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.583915    5043 manager.go:217] Machine: {Timestamp:2026-10-30 19:16:20.58339777 +0000 UTC m=+0.094457727 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3075596 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d309f39a27ba5e9f0e9491ff7e49a SystemUUID:ec2d309f-39a2-7ba5-e9f0-e9491ff7e49a BootID:15fc2b53-f375-489a-af3c-70aed27d98d1 Filesystems:[{Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/353ebfb4-7e11-41de-b992-277443f56960/volumes/kubernetes.io~projected/kube-api-access-2pfsl DeviceMajor:0 DeviceMinor:218 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d4e2c0ea2c6d38835a0854694a88a86923bd0ddbf7884508c97c088b14da698a/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/77fbf490-7af3-4bde-8222-4038e2219fff/volumes/kubernetes.io~projected/kube-api-access-l8ptq DeviceMajor:0 DeviceMinor:59 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9f302411-01be-438c-a30c-70c944afd630/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:214 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/765d8d9d-b89f-419f-b237-3f8034db1bed/volumes/kubernetes.io~projected/kube-api-access-njpgh DeviceMajor:0 DeviceMinor:222 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/247ff425-bed3-4297-b5c0-cc32b21bc179/volumes/kubernetes.io~projected/kube-api-access-28sgd DeviceMajor:0 DeviceMinor:225 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ca59d13272efc207f8bd73365bf08e0afe178dc4ec377b5ff544d3d9468374a7/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/853a64ec-147d-42eb-8946-e335de7878d3/volumes/kubernetes.io~projected/kube-api-access-7hx4s DeviceMajor:0 DeviceMinor:336 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-184 DeviceMajor:0 DeviceMinor:184 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/14b01e2ba44a03c9687bf7ae9ad7d318310a5472f8b4a12c835a6894a525714a/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a64211d0-1c48-41d2-b254-4ae84d5097d7/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:210 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c8faece2f3197dd85421545da5f23e9e2d01747c378c197afd296e70c026d12a/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dc581227-9542-4571-aced-63d191692a41/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bbe001d2613d79b8c9b2749f057a8249b324e97992746982fb7261e20a45458e/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0269ffa6-ce06-43f3-be2a-58f6aa3e6964/volumes/kubernetes.io~projected/kube-api-access-b6bsd DeviceMajor:0 DeviceMinor:224 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/103e94309533e5b3f462ba5781ecee4e8885c5ccf33c62bf1f560d1f596b95df/userdata/shm DeviceMajor:0 DeviceMinor:345 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/dc581227-9542-4571-aced-63d191692a41/volumes/kubernetes.io~projected/kube-api-access-4m9lt DeviceMajor:0 DeviceMinor:61 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/9f302411-01be-438c-a30c-70c944afd630/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:209 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/9f302411-01be-438c-a30c-70c944afd630/volumes/kubernetes.io~projected/kube-api-access-hpb5t DeviceMajor:0 DeviceMinor:223 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3367f10f4ffc027c5f101ffb7619c5ebe46b3e6ea784512fb62472c7e9c9fd71/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6ecb643b-1dce-4523-89df-93b22ad8ce6f/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/47e9832e-bc96-445a-a93e-9d24f84db3b3/volumes/kubernetes.io~projected/kube-api-access-6gnlf DeviceMajor:0 DeviceMinor:293 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ad4d9072a8cb0ef7d9197f2a1afef0c121f90b30eefd4f2ec780bfec7a063d4c/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f1ea27b2-80df-404c-ac39-460f495957b2/volumes/kubernetes.io~projected/kube-api-access-tngcd DeviceMajor:0 DeviceMinor:65 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/353ebfb4-7e11-41de-b992-277443f56960/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:205 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/e5002691-8257-4dab-a88b-3c1b60e8e6f8/volumes/kubernetes.io~projected/kube-api-access-9rz2d DeviceMajor:0 DeviceMinor:216 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/087138c34823747582e3393ac259cb9bb6078aad337a07f0de1831b5b7b1c0bf/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/06397fd4-3377-4eaa-8879-ebbb5943fbda/volumes/kubernetes.io~projected/kube-api-access-zh979 DeviceMajor:0 DeviceMinor:212 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aeef13dc-e06e-49c5-b893-64c7f73a7e44/volumes/kubernetes.io~projected/kube-api-access-tq4kp DeviceMajor:0 DeviceMinor:321 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/16705cc4-1ecc-47d1-9f4a-bb7eadc6294c/volumes/kubernetes.io~projected/kube-api-access-qr447 DeviceMajor:0 DeviceMinor:337 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/751934796f1ae1679278572b9ef2bb6bb010c50c66c514be4b59fa4e90ca381c/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/4dbf226e-336f-42c6-a766-cc0ddfe68bef/volumes/kubernetes.io~projected/kube-api-access-lbndd DeviceMajor:0 DeviceMinor:67 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/247ff425-bed3-4297-b5c0-cc32b21bc179/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:201 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dda4158a-327b-4b73-8673-3df1c1c0e400/volumes/kubernetes.io~projected/kube-api-access-m5c96 DeviceMajor:0 DeviceMinor:63 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/012f367ad67d33090ad1abbbf367716fe83ab03857802f0c325e08d1b002f679/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e5002691-8257-4dab-a88b-3c1b60e8e6f8/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:207 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/cbc4f7c7-bebd-4200-833e-1afbf3c2c81a/volumes/kubernetes.io~projected/kube-api-access-mcgjp DeviceMajor:0 DeviceMinor:226 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/a37bca55-21cf-4b76-9c05-e2c951647bf8/volumes/kubernetes.io~projected/kube-api-access-2vk66 DeviceMajor:0 DeviceMinor:351 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-181 DeviceMajor:0 DeviceMinor:181 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9f302411-01be-438c-a30c-70c944afd630/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:206 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c8d548b04e3d8e8c629730de6fc83598257db1ec337aabbb0514d1d0d007a668/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2268a2c217d2bf4d82f1c06fefc397fff88c6c383fddbdaa5c240ca4a9fdde56/userdata/shm DeviceMajor:0 DeviceMinor:356 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/a64211d0-1c48-41d2-b254-4ae84d5097d7/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:204 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/68201777-2ec5-43b2-b587-81e585865d08/volumes/kubernetes.io~projected/kube-api-access-s56kf DeviceMajor:0 DeviceMinor:220 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6331f51e-cf5e-4ba2-9eda-35fd62b14091/volumes/kubernetes.io~projected/kube-api-access-q5ldg DeviceMajor:0 DeviceMinor:62 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/2d261662-727d-4d68-9f4a-9790ecf16d46/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:64 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fe12e0dde11c16433babd16be15e78a1bce1c00b6854cf7addaa3503d314dd5a/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/487fa79e6ceaed2223a66678071091aae7f373b853288eb4e2de23616b76a96b/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/36ab52a2-65e2-4af6-a872-9052e12714d1/volumes/kubernetes.io~projected/kube-api-access-lksw9 DeviceMajor:0 DeviceMinor:58 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/765d8d9d-b89f-419f-b237-3f8034db1bed/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:203 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/247ff425-bed3-4297-b5c0-cc32b21bc179/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:211 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dc581227-9542-4571-aced-63d191692a41/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eab4e3466e8405b2944cdfca50c0dc8782020e6afc0d3ec51310419af686226e/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/68201777-2ec5-43b2-b587-81e585865d08/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:208 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/2d52a94d-72ac-4e5c-9472-104399a21a3b/volumes/kubernetes.io~projected/kube-api-access-zfglw DeviceMajor:0 DeviceMinor:219 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c7612191c1ecc48d5216a96ecbbc00bea70bdd93f79957dcfef867e1f359dbf9/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c7a27c1c553a6b4378a8b33e73553bea17b222b6c46e2fdf059823d42abb7f41/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fb1b480febdfaa3569526b9c3c98e490125407a1721d4f174e0eb8d77d707b74/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/a37bca55-21cf-4b76-9c05-e2c951647bf8/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:344 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/77fbf490-7af3-4bde-8222-4038e2219fff/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/a64211d0-1c48-41d2-b254-4ae84d5097d7/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:217 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/c84a5c22-7a46-4de1-aef7-32cb9a5453bb/volumes/kubernetes.io~projected/kube-api-access-zsjcc DeviceMajor:0 DeviceMinor:227 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/16a270602f3edfacb90114f8ac0b1d3a49a4c3dd5d021f56175b526b2136d7e2/userdata/shm DeviceMajor:0 DeviceMinor:79 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/76e21916-0744-4499-a0cb-64a7a44a8285/volumes/kubernetes.io~projected/kube-api-access-v6chk DeviceMajor:0 DeviceMinor:221 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ff2ebe84c24d4fd9301608b39c1fd681d19907a9cdea1f6c818f44762d03b0a9/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/00b97df0-857e-4ad9-b33a-4a59777b6f62/volumes/kubernetes.io~projected/kube-api-access-w8ww9 DeviceMajor:0 DeviceMinor:45 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/a64211d0-1c48-41d2-b254-4ae84d5097d7/volumes/kubernetes.io~projected/kube-api-access-6x795 DeviceMajor:0 DeviceMinor:213 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/c243f1d8-1c43-4eb8-93ec-cb28d72cc8a1/volumes/kubernetes.io~projected/kube-api-access-4tnx8 DeviceMajor:0 DeviceMinor:215 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/5338b461-6b7c-474b-8458-d1b53775ee0f/volumes/kubernetes.io~projected/kube-api-access-4xrrr DeviceMajor:0 DeviceMinor:333 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/128a8dad-66f8-414b-8c1d-14c7d3b58d25/volumes/kubernetes.io~projected/kube-api-access-jxkzk DeviceMajor:0 DeviceMinor:60 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/857baf1dd5d42d5472787a331e87333fc9cf66d405389472085f2803f7a4af2c/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/e8e297a2-a78a-4665-86d0-6d842bbcaa51/volumes/kubernetes.io~projected/kube-api-access-5xfxj DeviceMajor:0 DeviceMinor:66 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-81 DeviceMajor:0 DeviceMinor:81 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/76e21916-0744-4499-a0cb-64a7a44a8285/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b4cf0b0eda03520a427f16b5b09773069d86edc6466c727314bbc46f8bdfb62f/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5b80a342a983dcc8d8c93e5a7498eb7deed13dd3ae040b10a76c10a2c4c52644/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/77fbf490-7af3-4bde-8222-4038e2219fff/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9201f6f2-73b9-45cd-9950-3a07914d8ea9/volumes/kubernetes.io~projected/kube-api-access-ndw2r DeviceMajor:0 DeviceMinor:197 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7cf459bbec6deed978519f5d53895993cfcc83c8692fe905e753eb4122488bd9/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1b81426a4a1dfc6148525fe704c7fc993cfeb9bd6a25c91c4ba435fbafe15bce/userdata/shm DeviceMajor:0 DeviceMinor:342 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:012f367ad67d330 MacAddress:7e:74:f2:f0:68:64 Speed:10000 Mtu:8901} {Name:103e94309533e5b MacAddress:da:b2:db:a0:7e:dd Speed:10000 Mtu:8901} {Name:14b01e2ba44a03c MacAddress:ca:19:5b:2c:1a:1d Speed:10000 Mtu:8901} {Name:1b81426a4a1dfc6 MacAddress:b6:bf:10:91:7a:58 Speed:10000 Mtu:8901} {Name:2268a2c217d2bf4 MacAddress:8a:d1:d6:3e:15:96 Speed:10000 Mtu:8901} {Name:487fa79e6ceaed2 MacAddress:4a:5c:58:4b:4a:01 Speed:10000 Mtu:8901} {Name:5b80a342a983dcc MacAddress:c6:b3:20:cb:35:a6 Speed:10000 Mtu:8901} {Name:751934796f1ae16 MacAddress:a6:3b:65:c0:f6:60 Speed:10000 Mtu:8901} {Name:ad4d9072a8cb0ef MacAddress:f6:fb:8f:e4:03:a6 Speed:10000 Mtu:8901} {Name:b4cf0b0eda03520 MacAddress:d6:47:10:1d:0e:aa Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ef:b9:a9:bb Speed:0 Mtu:9001} {Name:br-int MacAddress:5a:d0:53:67:55:20 Speed:0 Mtu:8901} {Name:c7612191c1ecc48 MacAddress:5a:df:6e:88:fc:5f Speed:10000 Mtu:8901} {Name:c8d548b04e3d8e8 MacAddress:86:ae:47:db:63:de Speed:10000 Mtu:8901} {Name:c8faece2f3197dd MacAddress:8e:03:56:88:54:69 Speed:10000 Mtu:8901} {Name:ca59d13272efc20 MacAddress:b2:4e:30:5a:ff:fc Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:ef:b9:a9:bb Speed:0 Mtu:9001} {Name:fb1b480febdfaa3 MacAddress:16:ca:7f:0c:bc:f8 Speed:10000 Mtu:8901} {Name:fe12e0dde11c164 MacAddress:f2:5d:b0:2e:6e:e2 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:fe:dc:8c:88:5a:66 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ba:2e:24:4e:ba:00 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 19:16:20.584304 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.584307    5043 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 19:16:20.584607 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.584356    5043 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 19:16:20.584658 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.584631    5043 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 19:16:20.584862 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.584659    5043 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-84.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 19:16:20.584904 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.584864    5043 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 19:16:20.584904 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.584874    5043 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 19:16:20.584904 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.584884    5043 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:16:20.584977 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.584908    5043 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:16:20.584977 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.584927    5043 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 19:16:20.585092 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.585082    5043 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 19:16:20.585199 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.585190    5043 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 19:16:20.585242 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.585233    5043 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 19:16:20.585302 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.585248    5043 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 19:16:20.585302 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.585261    5043 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 19:16:20.585302 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.585274    5043 kubelet.go:405] "Adding apiserver pod source"
Oct 30 19:16:20.585302 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.585285    5043 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 19:16:20.586279 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.586262    5043 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 19:16:20.586279 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.586279    5043 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:16:20.586852 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.586828    5043 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 19:16:20.589065 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.589049    5043 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 19:16:20.589065 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.589062    5043 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:16:20.589177 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.589162    5043 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 19:16:20.589375 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.589363    5043 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 19:16:20.589408 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.589394    5043 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 19:16:20.589496 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.589487    5043 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 19:16:20.589535 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.589501    5043 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 19:16:20.589535 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.589508    5043 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 19:16:20.589535 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.589513    5043 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 19:16:20.589535 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.589520    5043 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 19:16:20.589535 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.589526    5043 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 19:16:20.589535 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.589532    5043 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 19:16:20.589781 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.589540    5043 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 19:16:20.589781 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.589551    5043 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 19:16:20.589781 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.589561    5043 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 19:16:20.589781 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.589578    5043 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 19:16:20.589781 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.589588    5043 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 19:16:20.589781 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.589616    5043 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 19:16:20.589781 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.589625    5043 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 19:16:20.593388 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.593374    5043 server.go:1263] "Started kubelet"
Oct 30 19:16:20.593922 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.593502    5043 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 19:16:20.593922 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.593505    5043 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 19:16:20.593922 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.593609    5043 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 19:16:20.594120 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.593926    5043 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 19:16:20.593947 ip-10-0-137-84 systemd[1]: Started Kubernetes Kubelet.
Oct 30 19:16:20.594965 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.594864    5043 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 19:16:20.599352 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.599332    5043 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 19:16:20.600269 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.599680    5043 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 19:16:20.600269 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.599696    5043 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 19:16:20.600269 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.599704    5043 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 19:16:20.600269 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:20.599807    5043 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-84.us-east-2.compute.internal\" not found"
Oct 30 19:16:20.600269 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.599841    5043 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 19:16:20.600269 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.600077    5043 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:10:40 +0000 UTC" deadline="2028-07-09 20:29:14.380436306 +0000 UTC"
Oct 30 19:16:20.600269 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.600110    5043 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14833h12m53.78032953s"
Oct 30 19:16:20.603184 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.603165    5043 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:16:20.604099 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.604078    5043 factory.go:153] Registering CRI-O factory
Oct 30 19:16:20.604194 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.604103    5043 factory.go:223] Registration of the crio container factory successfully
Oct 30 19:16:20.604246 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.604203    5043 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 19:16:20.604246 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.604214    5043 factory.go:55] Registering systemd factory
Oct 30 19:16:20.604246 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.604223    5043 factory.go:223] Registration of the systemd container factory successfully
Oct 30 19:16:20.604246 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.604226    5043 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:16:20.604246 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.604243    5043 factory.go:103] Registering Raw factory
Oct 30 19:16:20.604463 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.604260    5043 manager.go:1199] Started watching for new ooms in manager
Oct 30 19:16:20.604542 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.604526    5043 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:16:20.604587 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.604572    5043 manager.go:319] Starting recovery of all containers
Oct 30 19:16:20.604925 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.604854    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d261662-727d-4d68-9f4a-9790ecf16d46" volumeName="kubernetes.io/configmap/2d261662-727d-4d68-9f4a-9790ecf16d46-konnectivity-ca" seLinuxMountContext=""
Oct 30 19:16:20.605013 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.604940    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5338b461-6b7c-474b-8458-d1b53775ee0f" volumeName="kubernetes.io/projected/5338b461-6b7c-474b-8458-d1b53775ee0f-kube-api-access-4xrrr" seLinuxMountContext=""
Oct 30 19:16:20.605013 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.604957    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ecb643b-1dce-4523-89df-93b22ad8ce6f" volumeName="kubernetes.io/secret/6ecb643b-1dce-4523-89df-93b22ad8ce6f-original-pull-secret" seLinuxMountContext=""
Oct 30 19:16:20.605013 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.604968    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76e21916-0744-4499-a0cb-64a7a44a8285" volumeName="kubernetes.io/configmap/76e21916-0744-4499-a0cb-64a7a44a8285-config" seLinuxMountContext=""
Oct 30 19:16:20.605013 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.604982    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76e21916-0744-4499-a0cb-64a7a44a8285" volumeName="kubernetes.io/configmap/76e21916-0744-4499-a0cb-64a7a44a8285-trusted-ca" seLinuxMountContext=""
Oct 30 19:16:20.605013 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605001    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f302411-01be-438c-a30c-70c944afd630" volumeName="kubernetes.io/configmap/9f302411-01be-438c-a30c-70c944afd630-trusted-ca" seLinuxMountContext=""
Oct 30 19:16:20.605013 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605013    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c84a5c22-7a46-4de1-aef7-32cb9a5453bb" volumeName="kubernetes.io/projected/c84a5c22-7a46-4de1-aef7-32cb9a5453bb-kube-api-access-zsjcc" seLinuxMountContext=""
Oct 30 19:16:20.605409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605025    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a37bca55-21cf-4b76-9c05-e2c951647bf8" volumeName="kubernetes.io/secret/a37bca55-21cf-4b76-9c05-e2c951647bf8-signing-key" seLinuxMountContext=""
Oct 30 19:16:20.605409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605039    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a64211d0-1c48-41d2-b254-4ae84d5097d7" volumeName="kubernetes.io/configmap/a64211d0-1c48-41d2-b254-4ae84d5097d7-trusted-ca" seLinuxMountContext=""
Oct 30 19:16:20.605409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605050    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00b97df0-857e-4ad9-b33a-4a59777b6f62" volumeName="kubernetes.io/projected/00b97df0-857e-4ad9-b33a-4a59777b6f62-kube-api-access-w8ww9" seLinuxMountContext=""
Oct 30 19:16:20.605409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605062    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06397fd4-3377-4eaa-8879-ebbb5943fbda" volumeName="kubernetes.io/projected/06397fd4-3377-4eaa-8879-ebbb5943fbda-kube-api-access-zh979" seLinuxMountContext=""
Oct 30 19:16:20.605409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605073    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="247ff425-bed3-4297-b5c0-cc32b21bc179" volumeName="kubernetes.io/projected/247ff425-bed3-4297-b5c0-cc32b21bc179-kube-api-access-28sgd" seLinuxMountContext=""
Oct 30 19:16:20.605409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605086    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77fbf490-7af3-4bde-8222-4038e2219fff" volumeName="kubernetes.io/empty-dir/77fbf490-7af3-4bde-8222-4038e2219fff-etc-tuned" seLinuxMountContext=""
Oct 30 19:16:20.605409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605121    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f302411-01be-438c-a30c-70c944afd630" volumeName="kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-bound-sa-token" seLinuxMountContext=""
Oct 30 19:16:20.605409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605135    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aeef13dc-e06e-49c5-b893-64c7f73a7e44" volumeName="kubernetes.io/empty-dir/aeef13dc-e06e-49c5-b893-64c7f73a7e44-data-volume" seLinuxMountContext=""
Oct 30 19:16:20.605409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605147    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d261662-727d-4d68-9f4a-9790ecf16d46" volumeName="kubernetes.io/secret/2d261662-727d-4d68-9f4a-9790ecf16d46-agent-certs" seLinuxMountContext=""
Oct 30 19:16:20.605409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605164    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc581227-9542-4571-aced-63d191692a41" volumeName="kubernetes.io/configmap/dc581227-9542-4571-aced-63d191692a41-ovnkube-config" seLinuxMountContext=""
Oct 30 19:16:20.605409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605186    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5002691-8257-4dab-a88b-3c1b60e8e6f8" volumeName="kubernetes.io/empty-dir/e5002691-8257-4dab-a88b-3c1b60e8e6f8-tmp" seLinuxMountContext=""
Oct 30 19:16:20.605409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605197    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5002691-8257-4dab-a88b-3c1b60e8e6f8" volumeName="kubernetes.io/projected/e5002691-8257-4dab-a88b-3c1b60e8e6f8-kube-api-access-9rz2d" seLinuxMountContext=""
Oct 30 19:16:20.605409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605211    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5002691-8257-4dab-a88b-3c1b60e8e6f8" volumeName="kubernetes.io/secret/e5002691-8257-4dab-a88b-3c1b60e8e6f8-serving-cert" seLinuxMountContext=""
Oct 30 19:16:20.605409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605244    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36ab52a2-65e2-4af6-a872-9052e12714d1" volumeName="kubernetes.io/empty-dir/36ab52a2-65e2-4af6-a872-9052e12714d1-tmp-dir" seLinuxMountContext=""
Oct 30 19:16:20.605409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605257    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbc4f7c7-bebd-4200-833e-1afbf3c2c81a" volumeName="kubernetes.io/projected/cbc4f7c7-bebd-4200-833e-1afbf3c2c81a-kube-api-access-mcgjp" seLinuxMountContext=""
Oct 30 19:16:20.605409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605270    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc581227-9542-4571-aced-63d191692a41" volumeName="kubernetes.io/projected/dc581227-9542-4571-aced-63d191692a41-kube-api-access-4m9lt" seLinuxMountContext=""
Oct 30 19:16:20.605409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605283    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="765d8d9d-b89f-419f-b237-3f8034db1bed" volumeName="kubernetes.io/projected/765d8d9d-b89f-419f-b237-3f8034db1bed-kube-api-access-njpgh" seLinuxMountContext=""
Oct 30 19:16:20.605409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605295    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f302411-01be-438c-a30c-70c944afd630" volumeName="kubernetes.io/secret/9f302411-01be-438c-a30c-70c944afd630-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 19:16:20.605409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605308    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a64211d0-1c48-41d2-b254-4ae84d5097d7" volumeName="kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-bound-sa-token" seLinuxMountContext=""
Oct 30 19:16:20.605409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605328    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dda4158a-327b-4b73-8673-3df1c1c0e400" volumeName="kubernetes.io/projected/dda4158a-327b-4b73-8673-3df1c1c0e400-kube-api-access-m5c96" seLinuxMountContext=""
Oct 30 19:16:20.605409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605360    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5338b461-6b7c-474b-8458-d1b53775ee0f" volumeName="kubernetes.io/configmap/5338b461-6b7c-474b-8458-d1b53775ee0f-service-ca-bundle" seLinuxMountContext=""
Oct 30 19:16:20.605409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605379    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d52a94d-72ac-4e5c-9472-104399a21a3b" volumeName="kubernetes.io/projected/2d52a94d-72ac-4e5c-9472-104399a21a3b-kube-api-access-zfglw" seLinuxMountContext=""
Oct 30 19:16:20.605409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605392    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0269ffa6-ce06-43f3-be2a-58f6aa3e6964" volumeName="kubernetes.io/projected/0269ffa6-ce06-43f3-be2a-58f6aa3e6964-kube-api-access-b6bsd" seLinuxMountContext=""
Oct 30 19:16:20.605409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605405    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06397fd4-3377-4eaa-8879-ebbb5943fbda" volumeName="kubernetes.io/configmap/06397fd4-3377-4eaa-8879-ebbb5943fbda-telemetry-config" seLinuxMountContext=""
Oct 30 19:16:20.605409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605417    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="353ebfb4-7e11-41de-b992-277443f56960" volumeName="kubernetes.io/projected/353ebfb4-7e11-41de-b992-277443f56960-kube-api-access-2pfsl" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605430    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="353ebfb4-7e11-41de-b992-277443f56960" volumeName="kubernetes.io/secret/353ebfb4-7e11-41de-b992-277443f56960-serving-cert" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605443    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="853a64ec-147d-42eb-8946-e335de7878d3" volumeName="kubernetes.io/projected/853a64ec-147d-42eb-8946-e335de7878d3-kube-api-access-7hx4s" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605457    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5002691-8257-4dab-a88b-3c1b60e8e6f8" volumeName="kubernetes.io/configmap/e5002691-8257-4dab-a88b-3c1b60e8e6f8-service-ca-bundle" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605470    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0269ffa6-ce06-43f3-be2a-58f6aa3e6964" volumeName="kubernetes.io/empty-dir/0269ffa6-ce06-43f3-be2a-58f6aa3e6964-tmp-dir" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605482    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4dbf226e-336f-42c6-a766-cc0ddfe68bef" volumeName="kubernetes.io/configmap/4dbf226e-336f-42c6-a766-cc0ddfe68bef-multus-daemon-config" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605496    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68201777-2ec5-43b2-b587-81e585865d08" volumeName="kubernetes.io/configmap/68201777-2ec5-43b2-b587-81e585865d08-config" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605509    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f302411-01be-438c-a30c-70c944afd630" volumeName="kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-kube-api-access-hpb5t" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605522    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc581227-9542-4571-aced-63d191692a41" volumeName="kubernetes.io/configmap/dc581227-9542-4571-aced-63d191692a41-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605535    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dda4158a-327b-4b73-8673-3df1c1c0e400" volumeName="kubernetes.io/configmap/dda4158a-327b-4b73-8673-3df1c1c0e400-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605548    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76e21916-0744-4499-a0cb-64a7a44a8285" volumeName="kubernetes.io/projected/76e21916-0744-4499-a0cb-64a7a44a8285-kube-api-access-v6chk" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605560    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f302411-01be-438c-a30c-70c944afd630" volumeName="kubernetes.io/configmap/9f302411-01be-438c-a30c-70c944afd630-registry-certificates" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605573    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc581227-9542-4571-aced-63d191692a41" volumeName="kubernetes.io/secret/dc581227-9542-4571-aced-63d191692a41-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605585    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8e297a2-a78a-4665-86d0-6d842bbcaa51" volumeName="kubernetes.io/projected/e8e297a2-a78a-4665-86d0-6d842bbcaa51-kube-api-access-5xfxj" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605597    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1ea27b2-80df-404c-ac39-460f495957b2" volumeName="kubernetes.io/projected/f1ea27b2-80df-404c-ac39-460f495957b2-kube-api-access-tngcd" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605610    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="353ebfb4-7e11-41de-b992-277443f56960" volumeName="kubernetes.io/configmap/353ebfb4-7e11-41de-b992-277443f56960-config" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605622    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4dbf226e-336f-42c6-a766-cc0ddfe68bef" volumeName="kubernetes.io/projected/4dbf226e-336f-42c6-a766-cc0ddfe68bef-kube-api-access-lbndd" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605635    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68201777-2ec5-43b2-b587-81e585865d08" volumeName="kubernetes.io/projected/68201777-2ec5-43b2-b587-81e585865d08-kube-api-access-s56kf" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605648    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68201777-2ec5-43b2-b587-81e585865d08" volumeName="kubernetes.io/secret/68201777-2ec5-43b2-b587-81e585865d08-serving-cert" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605660    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f302411-01be-438c-a30c-70c944afd630" volumeName="kubernetes.io/secret/9f302411-01be-438c-a30c-70c944afd630-installation-pull-secrets" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605673    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a85cca66-a438-4117-9f50-977c7f528b7c" volumeName="kubernetes.io/configmap/a85cca66-a438-4117-9f50-977c7f528b7c-nginx-conf" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605687    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dda4158a-327b-4b73-8673-3df1c1c0e400" volumeName="kubernetes.io/configmap/dda4158a-327b-4b73-8673-3df1c1c0e400-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605700    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5002691-8257-4dab-a88b-3c1b60e8e6f8" volumeName="kubernetes.io/empty-dir/e5002691-8257-4dab-a88b-3c1b60e8e6f8-snapshots" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605714    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5338b461-6b7c-474b-8458-d1b53775ee0f" volumeName="kubernetes.io/configmap/5338b461-6b7c-474b-8458-d1b53775ee0f-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605727    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="247ff425-bed3-4297-b5c0-cc32b21bc179" volumeName="kubernetes.io/secret/247ff425-bed3-4297-b5c0-cc32b21bc179-default-certificate" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605776    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47e9832e-bc96-445a-a93e-9d24f84db3b3" volumeName="kubernetes.io/projected/47e9832e-bc96-445a-a93e-9d24f84db3b3-kube-api-access-6gnlf" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605789    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="765d8d9d-b89f-419f-b237-3f8034db1bed" volumeName="kubernetes.io/secret/765d8d9d-b89f-419f-b237-3f8034db1bed-cert" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605804    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f302411-01be-438c-a30c-70c944afd630" volumeName="kubernetes.io/empty-dir/9f302411-01be-438c-a30c-70c944afd630-ca-trust-extracted" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605817    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a37bca55-21cf-4b76-9c05-e2c951647bf8" volumeName="kubernetes.io/configmap/a37bca55-21cf-4b76-9c05-e2c951647bf8-signing-cabundle" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605830    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a37bca55-21cf-4b76-9c05-e2c951647bf8" volumeName="kubernetes.io/projected/a37bca55-21cf-4b76-9c05-e2c951647bf8-kube-api-access-2vk66" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605843    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c243f1d8-1c43-4eb8-93ec-cb28d72cc8a1" volumeName="kubernetes.io/projected/c243f1d8-1c43-4eb8-93ec-cb28d72cc8a1-kube-api-access-4tnx8" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605856    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="128a8dad-66f8-414b-8c1d-14c7d3b58d25" volumeName="kubernetes.io/projected/128a8dad-66f8-414b-8c1d-14c7d3b58d25-kube-api-access-jxkzk" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605870    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36ab52a2-65e2-4af6-a872-9052e12714d1" volumeName="kubernetes.io/projected/36ab52a2-65e2-4af6-a872-9052e12714d1-kube-api-access-lksw9" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605884    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a64211d0-1c48-41d2-b254-4ae84d5097d7" volumeName="kubernetes.io/secret/a64211d0-1c48-41d2-b254-4ae84d5097d7-installation-pull-secrets" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605910    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aeef13dc-e06e-49c5-b893-64c7f73a7e44" volumeName="kubernetes.io/projected/aeef13dc-e06e-49c5-b893-64c7f73a7e44-kube-api-access-tq4kp" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605925    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbc4f7c7-bebd-4200-833e-1afbf3c2c81a" volumeName="kubernetes.io/configmap/cbc4f7c7-bebd-4200-833e-1afbf3c2c81a-samples-operator-config" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605938    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="247ff425-bed3-4297-b5c0-cc32b21bc179" volumeName="kubernetes.io/secret/247ff425-bed3-4297-b5c0-cc32b21bc179-stats-auth" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605951    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6331f51e-cf5e-4ba2-9eda-35fd62b14091" volumeName="kubernetes.io/configmap/6331f51e-cf5e-4ba2-9eda-35fd62b14091-iptables-alerter-script" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605964    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77fbf490-7af3-4bde-8222-4038e2219fff" volumeName="kubernetes.io/empty-dir/77fbf490-7af3-4bde-8222-4038e2219fff-tmp" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605978    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9201f6f2-73b9-45cd-9950-3a07914d8ea9" volumeName="kubernetes.io/projected/9201f6f2-73b9-45cd-9950-3a07914d8ea9-kube-api-access-ndw2r" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.605990    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a64211d0-1c48-41d2-b254-4ae84d5097d7" volumeName="kubernetes.io/configmap/a64211d0-1c48-41d2-b254-4ae84d5097d7-registry-certificates" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.606004    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a64211d0-1c48-41d2-b254-4ae84d5097d7" volumeName="kubernetes.io/empty-dir/a64211d0-1c48-41d2-b254-4ae84d5097d7-ca-trust-extracted" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.606016    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dda4158a-327b-4b73-8673-3df1c1c0e400" volumeName="kubernetes.io/configmap/dda4158a-327b-4b73-8673-3df1c1c0e400-cni-binary-copy" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.606029    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a64211d0-1c48-41d2-b254-4ae84d5097d7" volumeName="kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-kube-api-access-6x795" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.606043    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0269ffa6-ce06-43f3-be2a-58f6aa3e6964" volumeName="kubernetes.io/configmap/0269ffa6-ce06-43f3-be2a-58f6aa3e6964-config-volume" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.606055    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6331f51e-cf5e-4ba2-9eda-35fd62b14091" volumeName="kubernetes.io/projected/6331f51e-cf5e-4ba2-9eda-35fd62b14091-kube-api-access-q5ldg" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.606068    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76e21916-0744-4499-a0cb-64a7a44a8285" volumeName="kubernetes.io/secret/76e21916-0744-4499-a0cb-64a7a44a8285-serving-cert" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.606081    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77fbf490-7af3-4bde-8222-4038e2219fff" volumeName="kubernetes.io/projected/77fbf490-7af3-4bde-8222-4038e2219fff-kube-api-access-l8ptq" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.606093    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc581227-9542-4571-aced-63d191692a41" volumeName="kubernetes.io/configmap/dc581227-9542-4571-aced-63d191692a41-env-overrides" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.606105    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1ea27b2-80df-404c-ac39-460f495957b2" volumeName="kubernetes.io/configmap/f1ea27b2-80df-404c-ac39-460f495957b2-serviceca" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.606118    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16705cc4-1ecc-47d1-9f4a-bb7eadc6294c" volumeName="kubernetes.io/projected/16705cc4-1ecc-47d1-9f4a-bb7eadc6294c-kube-api-access-qr447" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.606131    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4dbf226e-336f-42c6-a766-cc0ddfe68bef" volumeName="kubernetes.io/configmap/4dbf226e-336f-42c6-a766-cc0ddfe68bef-cni-binary-copy" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.606144    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5338b461-6b7c-474b-8458-d1b53775ee0f" volumeName="kubernetes.io/empty-dir/5338b461-6b7c-474b-8458-d1b53775ee0f-archives-path" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.606156    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a64211d0-1c48-41d2-b254-4ae84d5097d7" volumeName="kubernetes.io/secret/a64211d0-1c48-41d2-b254-4ae84d5097d7-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.606169    5043 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5002691-8257-4dab-a88b-3c1b60e8e6f8" volumeName="kubernetes.io/configmap/e5002691-8257-4dab-a88b-3c1b60e8e6f8-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.606183    5043 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 19:16:20.606645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.606191    5043 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 19:16:20.610067 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.610039    5043 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 19:16:20.621808 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.621781    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cmw6w_76e21916-0744-4499-a0cb-64a7a44a8285/console-operator/1.log"
Oct 30 19:16:20.623351 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.623321    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cmw6w_76e21916-0744-4499-a0cb-64a7a44a8285/console-operator/1.log"
Oct 30 19:16:20.635174 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.635119    5043 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 19:16:20.636675 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.636656    5043 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 19:16:20.636675 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.636675    5043 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 19:16:20.636847 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.636699    5043 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 19:16:20.636847 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:20.636763    5043 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 19:16:20.640944 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.640907    5043 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:16:20.662204 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.662142    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cmw6w_76e21916-0744-4499-a0cb-64a7a44a8285/console-operator/1.log"
Oct 30 19:16:20.662204 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.662171    5043 generic.go:356] "Generic (PLEG): container finished" podID="76e21916-0744-4499-a0cb-64a7a44a8285" containerID="b3f38d74f7964cb2e0db345a5bf20e6f2b18ed1ffffa37aafe2a406f3f91103c" exitCode=255
Oct 30 19:16:20.668819 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.668790    5043 generic.go:356] "Generic (PLEG): container finished" podID="dda4158a-327b-4b73-8673-3df1c1c0e400" containerID="3f3f5c8ef800f8a55e6a657f9c0ed27859a6d16ecde03aa60107aa15a8896659" exitCode=0
Oct 30 19:16:20.668819 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.668813    5043 generic.go:356] "Generic (PLEG): container finished" podID="dda4158a-327b-4b73-8673-3df1c1c0e400" containerID="8a63990b0547bab66217643184c8411c235ef64fe4168470f0c355603bee1d79" exitCode=0
Oct 30 19:16:20.668819 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.668821    5043 generic.go:356] "Generic (PLEG): container finished" podID="dda4158a-327b-4b73-8673-3df1c1c0e400" containerID="783623f779ea310dd77c989130e4752363898368076cbcdb419bc0062c39ce81" exitCode=0
Oct 30 19:16:20.668819 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.668826    5043 generic.go:356] "Generic (PLEG): container finished" podID="dda4158a-327b-4b73-8673-3df1c1c0e400" containerID="6274a8be63c45b6c33ce5b00999aba3dcf4d72baa2f432e9a4b929b38aefda61" exitCode=0
Oct 30 19:16:20.669125 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.668838    5043 generic.go:356] "Generic (PLEG): container finished" podID="dda4158a-327b-4b73-8673-3df1c1c0e400" containerID="0daaa62fd12d61178eea10740d73a5cf9f347ed5bde484bf1ca2862e3dacea91" exitCode=0
Oct 30 19:16:20.669125 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.668847    5043 generic.go:356] "Generic (PLEG): container finished" podID="dda4158a-327b-4b73-8673-3df1c1c0e400" containerID="9ad16e4217e5ee2917024c4b21645d097d35b4e15156d2a54f77067cd32726e1" exitCode=0
Oct 30 19:16:20.684113 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.684071    5043 generic.go:356] "Generic (PLEG): container finished" podID="2e49edc306f78a76399f3059c085e607" containerID="ce2eb36c5f6cbceb55fe5ed767c9c6692e1ba7a8da3637235b0de54f5929cf7a" exitCode=0
Oct 30 19:16:20.704094 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.704072    5043 manager.go:324] Recovery completed
Oct 30 19:16:20.724339 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.724310    5043 cpu_manager.go:226] "Starting" policy="none"
Oct 30 19:16:20.724339 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.724327    5043 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 19:16:20.724556 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.724350    5043 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 19:16:20.724556 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.724502    5043 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 19:16:20.724556 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.724517    5043 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 19:16:20.724556 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.724541    5043 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 19:16:20.724556 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.724550    5043 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 19:16:20.724556 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.724557    5043 policy_none.go:50] "Start"
Oct 30 19:16:20.724860 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.724567    5043 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 19:16:20.724860 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.724580    5043 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 19:16:20.724860 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.724791    5043 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 19:16:20.724860 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.724803    5043 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 19:16:20.724860 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.724812    5043 policy_none.go:44] "Start"
Oct 30 19:16:20.731275 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.731242    5043 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 19:16:20.734504 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.734486    5043 manager.go:341] "Starting Device Plugin manager"
Oct 30 19:16:20.735256 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:20.734520    5043 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 19:16:20.735302 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.735265    5043 server.go:87] "Starting device plugin registration server"
Oct 30 19:16:20.735613 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.735600    5043 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 19:16:20.735643 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.735618    5043 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 19:16:20.735752 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.735682    5043 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 19:16:20.735868 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.735837    5043 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 19:16:20.735923 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.735883    5043 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 19:16:20.735923 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.735890    5043 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 19:16:20.736026 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.735938    5043 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:16:20.735867301Z","UUID":"f08832d8-f7db-4f45-8571-7d6412d2b695","Handler":null,"Name":"","Endpoint":""}
Oct 30 19:16:20.736830 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.736797    5043 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-84.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-84.us-east-2.compute.internal"]
Oct 30 19:16:20.737453 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.737420    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-84.us-east-2.compute.internal" event={"ID":"532f06e8e868303658c6aeb582644516","Type":"ContainerStarted","Data":"2c88a6c99ffe65e37e10acbb712fcdd04dd9da8a7296037eca33248a0774a0e6"}
Oct 30 19:16:20.737540 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.737461    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-84.us-east-2.compute.internal" event={"ID":"532f06e8e868303658c6aeb582644516","Type":"ContainerStarted","Data":"3367f10f4ffc027c5f101ffb7619c5ebe46b3e6ea784512fb62472c7e9c9fd71"}
Oct 30 19:16:20.737540 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.737512    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-84.us-east-2.compute.internal" event={"ID":"2e49edc306f78a76399f3059c085e607","Type":"ContainerStarted","Data":"bf057eaa911bd9bf1028bfb2ba7b4b84535a469b560e362370dec8b7c4c0fa8d"}
Oct 30 19:16:20.737540 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.737531    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-84.us-east-2.compute.internal" event={"ID":"2e49edc306f78a76399f3059c085e607","Type":"ContainerDied","Data":"ce2eb36c5f6cbceb55fe5ed767c9c6692e1ba7a8da3637235b0de54f5929cf7a"}
Oct 30 19:16:20.737678 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.737550    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-84.us-east-2.compute.internal" event={"ID":"2e49edc306f78a76399f3059c085e607","Type":"ContainerStarted","Data":"d4e2c0ea2c6d38835a0854694a88a86923bd0ddbf7884508c97c088b14da698a"}
Oct 30 19:16:20.737846 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.737827    5043 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-84.us-east-2.compute.internal"
Oct 30 19:16:20.737909 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:20.737887    5043 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 19:16:20.738065 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.738046    5043 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-84.us-east-2.compute.internal"
Oct 30 19:16:20.738325 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.738300    5043 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 19:16:20.738414 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.738349    5043 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 19:16:20.751753 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.751711    5043 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:16:20.751913 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:20.751768    5043 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-84.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-84.us-east-2.compute.internal"
Oct 30 19:16:20.752186 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.752172    5043 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:16:20.752258 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:20.752194    5043 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-84.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-84.us-east-2.compute.internal"
Oct 30 19:16:20.810849 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.810809    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2e49edc306f78a76399f3059c085e607-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-84.us-east-2.compute.internal\" (UID: \"2e49edc306f78a76399f3059c085e607\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-84.us-east-2.compute.internal"
Oct 30 19:16:20.811004 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.810857    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e49edc306f78a76399f3059c085e607-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-84.us-east-2.compute.internal\" (UID: \"2e49edc306f78a76399f3059c085e607\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-84.us-east-2.compute.internal"
Oct 30 19:16:20.811004 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.810878    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/532f06e8e868303658c6aeb582644516-config\") pod \"kube-apiserver-proxy-ip-10-0-137-84.us-east-2.compute.internal\" (UID: \"532f06e8e868303658c6aeb582644516\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-84.us-east-2.compute.internal"
Oct 30 19:16:20.838927 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.838889    5043 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:16:20.840268 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.840245    5043 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-84.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:16:20.840397 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.840285    5043 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-84.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:16:20.840397 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.840301    5043 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-84.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:16:20.840397 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.840365    5043 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-84.us-east-2.compute.internal"
Oct 30 19:16:20.852978 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.852957    5043 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-137-84.us-east-2.compute.internal"
Oct 30 19:16:20.853087 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.853017    5043 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-84.us-east-2.compute.internal"
Oct 30 19:16:20.911192 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.911155    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2e49edc306f78a76399f3059c085e607-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-84.us-east-2.compute.internal\" (UID: \"2e49edc306f78a76399f3059c085e607\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-84.us-east-2.compute.internal"
Oct 30 19:16:20.911192 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.911192    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e49edc306f78a76399f3059c085e607-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-84.us-east-2.compute.internal\" (UID: \"2e49edc306f78a76399f3059c085e607\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-84.us-east-2.compute.internal"
Oct 30 19:16:20.911382 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.911212    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/532f06e8e868303658c6aeb582644516-config\") pod \"kube-apiserver-proxy-ip-10-0-137-84.us-east-2.compute.internal\" (UID: \"532f06e8e868303658c6aeb582644516\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-84.us-east-2.compute.internal"
Oct 30 19:16:20.911382 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.911267    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2e49edc306f78a76399f3059c085e607-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-84.us-east-2.compute.internal\" (UID: \"2e49edc306f78a76399f3059c085e607\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-84.us-east-2.compute.internal"
Oct 30 19:16:20.911382 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.911270    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/532f06e8e868303658c6aeb582644516-config\") pod \"kube-apiserver-proxy-ip-10-0-137-84.us-east-2.compute.internal\" (UID: \"532f06e8e868303658c6aeb582644516\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-84.us-east-2.compute.internal"
Oct 30 19:16:20.911382 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:20.911272    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e49edc306f78a76399f3059c085e607-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-84.us-east-2.compute.internal\" (UID: \"2e49edc306f78a76399f3059c085e607\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-84.us-east-2.compute.internal"
Oct 30 19:16:21.585978 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.585942    5043 apiserver.go:52] "Watching apiserver"
Oct 30 19:16:21.599870 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.599845    5043 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 19:16:21.600568 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.600540    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vv4df","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-62bhw","openshift-ingress-canary/ingress-canary-9jkcb","openshift-service-ca-operator/service-ca-operator-544d749c8c-vd8tj","openshift-image-registry/image-registry-748f8689f4-6lwjn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-84.us-east-2.compute.internal","openshift-dns/dns-default-gkn6q","openshift-monitoring/cluster-monitoring-operator-884576c57-2sdkw","openshift-cluster-node-tuning-operator/tuned-tkdbv","kube-system/konnectivity-agent-lqmqr","openshift-multus/network-metrics-daemon-n9cws","kube-system/global-pull-secret-syncer-wg2df","openshift-multus/multus-jxqzn","openshift-service-ca/service-ca-5fffc97bdf-b8cs2","openshift-image-registry/image-registry-78dc458d47-tqn8s","openshift-network-diagnostics/network-check-source-6f78d59f99-bvcb4","openshift-image-registry/node-ca-zj8d6","openshift-insights/insights-runtime-extractor-smgf6","openshift-dns/node-resolver-xjjqh","kube-system/kube-apiserver-proxy-ip-10-0-137-84.us-east-2.compute.internal","openshift-insights/periodic-gathering-2lcwk-gjv8f","openshift-network-console/networking-console-plugin-86c478bb88-lpg7w","openshift-network-diagnostics/network-check-target-n9zx2","openshift-ingress/router-default-75866dc98c-4mnhf","openshift-insights/insights-operator-7c4986d449-fz9hf","kube-system/kas-connection-checker-685895c7c9-zqvrw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr","openshift-kube-storage-version-migrator/migrator-66df5d496b-qggf8","kube-system/kas-connection-checker-685895c7c9-6b8sd","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p5nzf","openshift-network-operator/iptables-alerter-26xq4","openshift-ovn-kubernetes/ovnkube-node-s8t94","openshift-multus/multus-additional-cni-plugins-w2262","openshift-console-operator/console-operator-7cd56b8dbc-cmw6w","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-mg98j","kube-system/kas-connection-checker-685895c7c9-65kc2"]
Oct 30 19:16:21.602792 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.602768    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-75866dc98c-4mnhf"
Oct 30 19:16:21.603475 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.603453    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:21.603851 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.603828    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p5nzf"
Oct 30 19:16:21.604630 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.604610    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2sdkw"
Oct 30 19:16:21.605437 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.605420    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-lpg7w"
Oct 30 19:16:21.605656 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.605630    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 19:16:21.605840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.605825    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 19:16:21.607502 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.606315    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-pzpjq\""
Oct 30 19:16:21.607502 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.606770    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 19:16:21.607502 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.607398    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-fnphj\""
Oct 30 19:16:21.607702 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.607562    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 19:16:21.607702 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.607580    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-fdz8t\""
Oct 30 19:16:21.607811 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.607718    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wqq7d\""
Oct 30 19:16:21.607964 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.607942    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-xkqpk\""
Oct 30 19:16:21.607964 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.607963    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-fl6tr\""
Oct 30 19:16:21.608113 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.607994    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 19:16:21.608228 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.608211    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-w9tb6\""
Oct 30 19:16:21.608485 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.608468    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 19:16:21.608678 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.608663    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 19:16:21.608801 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.608786    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-xx4ns\""
Oct 30 19:16:21.609115 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.609098    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 19:16:21.609295 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.609277    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 19:16:21.609360 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.609304    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 19:16:21.609476 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.609458    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-bb48c\""
Oct 30 19:16:21.609706 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.609690    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 19:16:21.609784 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.609771    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-c2vmx\""
Oct 30 19:16:21.609829 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.609809    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 19:16:21.609967 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.609951    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 19:16:21.610090 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.610063    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 19:16:21.610270 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.610220    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:16:21.610664 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.610638    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 19:16:21.610664 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.610660    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 19:16:21.611698 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.611673    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 19:16:21.611698 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.611695    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 19:16:21.611849 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.611720    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 19:16:21.611849 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.611761    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 19:16:21.611849 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.611787    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 19:16:21.611849 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.611835    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 19:16:21.612040 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.611852    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 19:16:21.612040 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.611862    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 19:16:21.612040 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.611889    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 19:16:21.612040 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.611899    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 19:16:21.612040 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.611916    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 19:16:21.612319 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.612030    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 19:16:21.612319 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.612062    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 19:16:21.612319 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.612069    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:16:21.612319 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.612102    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:16:21.612319 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.612126    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 19:16:21.612319 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.612148    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 19:16:21.612319 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.612194    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 19:16:21.612505 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.612422    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 19:16:21.612505 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.612486    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 19:16:21.613008 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.612945    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 19:16:21.613008 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.612956    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:16:21.613136 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.613021    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 19:16:21.613136 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.613100    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 19:16:21.613136 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.613111    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:16:21.613247 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.613154    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 19:16:21.613247 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.613162    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-4rpsp\""
Oct 30 19:16:21.613247 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.613222    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7nrdg\""
Oct 30 19:16:21.613339 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.613288    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 19:16:21.614380 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.614363    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 19:16:21.614468 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.614396    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 19:16:21.615635 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.615617    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:21.616754 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.616717    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 19:16:21.617371 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.617355    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9cws"
Oct 30 19:16:21.618127 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.618110    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gkn6q"
Oct 30 19:16:21.619497 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.619479    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-smgf6"
Oct 30 19:16:21.626366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.622262    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 19:16:21.626366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.622283    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 19:16:21.626366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.622374    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 19:16:21.626366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.622493    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 19:16:21.626366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.622553    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 19:16:21.626366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.622680    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 19:16:21.626366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.622690    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 19:16:21.626366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.622920    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 19:16:21.626366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.623056    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 19:16:21.626366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.623312    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qmpzj\""
Oct 30 19:16:21.626366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.623420    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9tgzd\""
Oct 30 19:16:21.626366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.623426    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:16:21.626366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.623912    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 19:16:21.626366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.623920    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 19:16:21.626366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.623952    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 19:16:21.626366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.623999    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-n7s42\""
Oct 30 19:16:21.626366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.624151    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 19:16:21.626366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.624185    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 19:16:21.626366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.624240    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 19:16:21.626366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.624299    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-6bckm\""
Oct 30 19:16:21.626366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.624416    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jzjv5\""
Oct 30 19:16:21.626366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.624488    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 19:16:21.626366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.624623    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 19:16:21.626366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.624669    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 19:16:21.626366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.624689    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-2qhvn\""
Oct 30 19:16:21.626366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.624761    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:16:21.626366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.624788    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 19:16:21.626366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.624807    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 19:16:21.626366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.624811    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-wctlt\""
Oct 30 19:16:21.626366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.624990    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 19:16:21.626366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.625026    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 19:16:21.626366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.625049    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7b7kv\""
Oct 30 19:16:21.626366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.625224    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-fppr2\""
Oct 30 19:16:21.626366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.625294    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xph5z\""
Oct 30 19:16:21.626366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.625316    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 19:16:21.626366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.625392    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 19:16:21.626366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.625514    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mglp5\""
Oct 30 19:16:21.626366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.625595    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 19:16:21.626366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.625654    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 19:16:21.626366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.625838    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 19:16:21.626366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.625965    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 19:16:21.626366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.625992    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lfmhh\""
Oct 30 19:16:21.626366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.625318    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-v6xn8\""
Oct 30 19:16:21.626366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.626161    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-bfjrl\""
Oct 30 19:16:21.626366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.626184    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 19:16:21.628033 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.626686    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 19:16:21.628033 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.626772    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 19:16:21.628033 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.626835    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 19:16:21.628033 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.626865    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 19:16:21.628033 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.626983    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2k8z2\""
Oct 30 19:16:21.629384 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.629367    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 19:16:21.629527 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.629511    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 19:16:21.629616 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.629605    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-46v7v\""
Oct 30 19:16:21.629616 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.629606    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 19:16:21.629714 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.629641    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 19:16:21.702052 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.702026    5043 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 19:16:21.714439 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.714408    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dda4158a-327b-4b73-8673-3df1c1c0e400-system-cni-dir\") pod \"multus-additional-cni-plugins-w2262\" (UID: \"dda4158a-327b-4b73-8673-3df1c1c0e400\") " pod="openshift-multus/multus-additional-cni-plugins-w2262"
Oct 30 19:16:21.714439 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.714440    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cbc4f7c7-bebd-4200-833e-1afbf3c2c81a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-p5nzf\" (UID: \"cbc4f7c7-bebd-4200-833e-1afbf3c2c81a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p5nzf"
Oct 30 19:16:21.714610 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.714458    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-host-var-lib-cni-bin\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:16:21.714610 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.714476    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-node-log\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.714610 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.714528    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0269ffa6-ce06-43f3-be2a-58f6aa3e6964-config-volume\") pod \"dns-default-gkn6q\" (UID: \"0269ffa6-ce06-43f3-be2a-58f6aa3e6964\") " pod="openshift-dns/dns-default-gkn6q"
Oct 30 19:16:21.714610 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.714568    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dda4158a-327b-4b73-8673-3df1c1c0e400-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w2262\" (UID: \"dda4158a-327b-4b73-8673-3df1c1c0e400\") " pod="openshift-multus/multus-additional-cni-plugins-w2262"
Oct 30 19:16:21.714610 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.714590    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-host-var-lib-kubelet\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:16:21.714867 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.714673    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/353ebfb4-7e11-41de-b992-277443f56960-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-62bhw\" (UID: \"353ebfb4-7e11-41de-b992-277443f56960\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-62bhw"
Oct 30 19:16:21.714867 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.714712    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a85cca66-a438-4117-9f50-977c7f528b7c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lpg7w\" (UID: \"a85cca66-a438-4117-9f50-977c7f528b7c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lpg7w"
Oct 30 19:16:21.714867 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.714783    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-host\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:16:21.714867 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.714812    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/36ab52a2-65e2-4af6-a872-9052e12714d1-hosts-file\") pod \"node-resolver-xjjqh\" (UID: \"36ab52a2-65e2-4af6-a872-9052e12714d1\") " pod="openshift-dns/node-resolver-xjjqh"
Oct 30 19:16:21.714867 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.714841    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/128a8dad-66f8-414b-8c1d-14c7d3b58d25-etc-selinux\") pod \"aws-ebs-csi-driver-node-6k5kr\" (UID: \"128a8dad-66f8-414b-8c1d-14c7d3b58d25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr"
Oct 30 19:16:21.715078 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.714871    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0269ffa6-ce06-43f3-be2a-58f6aa3e6964-config-volume\") pod \"dns-default-gkn6q\" (UID: \"0269ffa6-ce06-43f3-be2a-58f6aa3e6964\") " pod="openshift-dns/dns-default-gkn6q"
Oct 30 19:16:21.715078 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.714885    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-etc-kubernetes\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:16:21.715078 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.714923    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36ab52a2-65e2-4af6-a872-9052e12714d1-tmp-dir\") pod \"node-resolver-xjjqh\" (UID: \"36ab52a2-65e2-4af6-a872-9052e12714d1\") " pod="openshift-dns/node-resolver-xjjqh"
Oct 30 19:16:21.715078 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.714939    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/353ebfb4-7e11-41de-b992-277443f56960-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-62bhw\" (UID: \"353ebfb4-7e11-41de-b992-277443f56960\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-62bhw"
Oct 30 19:16:21.715078 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.714959    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr447\" (UniqueName: \"kubernetes.io/projected/16705cc4-1ecc-47d1-9f4a-bb7eadc6294c-kube-api-access-qr447\") pod \"migrator-66df5d496b-vv4df\" (UID: \"16705cc4-1ecc-47d1-9f4a-bb7eadc6294c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vv4df"
Oct 30 19:16:21.715078 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.715000    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36ab52a2-65e2-4af6-a872-9052e12714d1-tmp-dir\") pod \"node-resolver-xjjqh\" (UID: \"36ab52a2-65e2-4af6-a872-9052e12714d1\") " pod="openshift-dns/node-resolver-xjjqh"
Oct 30 19:16:21.715078 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.714999    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dda4158a-327b-4b73-8673-3df1c1c0e400-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w2262\" (UID: \"dda4158a-327b-4b73-8673-3df1c1c0e400\") " pod="openshift-multus/multus-additional-cni-plugins-w2262"
Oct 30 19:16:21.715078 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.715029    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dc581227-9542-4571-aced-63d191692a41-ovnkube-script-lib\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.715078 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.715056    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a64211d0-1c48-41d2-b254-4ae84d5097d7-trusted-ca\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:21.715078 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.715072    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-etc-sysctl-conf\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:16:21.715536 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.715086    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-etc-systemd\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:16:21.715536 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.715104    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/77fbf490-7af3-4bde-8222-4038e2219fff-etc-tuned\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:16:21.715536 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.715150    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hx4s\" (UniqueName: \"kubernetes.io/projected/853a64ec-147d-42eb-8946-e335de7878d3-kube-api-access-7hx4s\") pod \"migrator-66df5d496b-qggf8\" (UID: \"853a64ec-147d-42eb-8946-e335de7878d3\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qggf8"
Oct 30 19:16:21.715536 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.715212    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcgjp\" (UniqueName: \"kubernetes.io/projected/cbc4f7c7-bebd-4200-833e-1afbf3c2c81a-kube-api-access-mcgjp\") pod \"cluster-samples-operator-6ddb6c8b94-p5nzf\" (UID: \"cbc4f7c7-bebd-4200-833e-1afbf3c2c81a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p5nzf"
Oct 30 19:16:21.715536 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.715225    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/77fbf490-7af3-4bde-8222-4038e2219fff-etc-tuned\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:16:21.715536 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.715241    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-host-kubelet\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.715536 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.715399    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a64211d0-1c48-41d2-b254-4ae84d5097d7-trusted-ca\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:21.715536 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.715444    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/dc581227-9542-4571-aced-63d191692a41-ovnkube-script-lib\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.715536 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.715463    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e5002691-8257-4dab-a88b-3c1b60e8e6f8-tmp\") pod \"insights-operator-7c4986d449-fz9hf\" (UID: \"e5002691-8257-4dab-a88b-3c1b60e8e6f8\") " pod="openshift-insights/insights-operator-7c4986d449-fz9hf"
Oct 30 19:16:21.715536 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.715517    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e5002691-8257-4dab-a88b-3c1b60e8e6f8-tmp\") pod \"insights-operator-7c4986d449-fz9hf\" (UID: \"e5002691-8257-4dab-a88b-3c1b60e8e6f8\") " pod="openshift-insights/insights-operator-7c4986d449-fz9hf"
Oct 30 19:16:21.716089 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.715573    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/128a8dad-66f8-414b-8c1d-14c7d3b58d25-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6k5kr\" (UID: \"128a8dad-66f8-414b-8c1d-14c7d3b58d25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr"
Oct 30 19:16:21.716089 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.715626    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/128a8dad-66f8-414b-8c1d-14c7d3b58d25-sys-fs\") pod \"aws-ebs-csi-driver-node-6k5kr\" (UID: \"128a8dad-66f8-414b-8c1d-14c7d3b58d25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr"
Oct 30 19:16:21.716089 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.715650    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-etc-sysctl-d\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:16:21.716089 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.715675    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9f302411-01be-438c-a30c-70c944afd630-ca-trust-extracted\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:21.716089 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.715701    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xfxj\" (UniqueName: \"kubernetes.io/projected/e8e297a2-a78a-4665-86d0-6d842bbcaa51-kube-api-access-5xfxj\") pod \"network-metrics-daemon-n9cws\" (UID: \"e8e297a2-a78a-4665-86d0-6d842bbcaa51\") " pod="openshift-multus/network-metrics-daemon-n9cws"
Oct 30 19:16:21.716089 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.715759    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq4kp\" (UniqueName: \"kubernetes.io/projected/aeef13dc-e06e-49c5-b893-64c7f73a7e44-kube-api-access-tq4kp\") pod \"insights-runtime-extractor-smgf6\" (UID: \"aeef13dc-e06e-49c5-b893-64c7f73a7e44\") " pod="openshift-insights/insights-runtime-extractor-smgf6"
Oct 30 19:16:21.716089 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.715787    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9f302411-01be-438c-a30c-70c944afd630-ca-trust-extracted\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:21.716089 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.715797    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-var-lib-openvswitch\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.716089 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.715836    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-run-openvswitch\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.716089 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.715864    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-run-ovn\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.716089 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.715892    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-multus-socket-dir-parent\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:16:21.716089 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.715925    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-multus-conf-dir\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:16:21.716089 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.715952    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/247ff425-bed3-4297-b5c0-cc32b21bc179-metrics-certs\") pod \"router-default-75866dc98c-4mnhf\" (UID: \"247ff425-bed3-4297-b5c0-cc32b21bc179\") " pod="openshift-ingress/router-default-75866dc98c-4mnhf"
Oct 30 19:16:21.716089 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.715985    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a85cca66-a438-4117-9f50-977c7f528b7c-nginx-conf\") pod \"networking-console-plugin-86c478bb88-lpg7w\" (UID: \"a85cca66-a438-4117-9f50-977c7f528b7c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lpg7w"
Oct 30 19:16:21.716089 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.716032    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/247ff425-bed3-4297-b5c0-cc32b21bc179-service-ca-bundle\") pod \"router-default-75866dc98c-4mnhf\" (UID: \"247ff425-bed3-4297-b5c0-cc32b21bc179\") " pod="openshift-ingress/router-default-75866dc98c-4mnhf"
Oct 30 19:16:21.716834 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.716107    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m9lt\" (UniqueName: \"kubernetes.io/projected/dc581227-9542-4571-aced-63d191692a41-kube-api-access-4m9lt\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.716834 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.716134    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-registry-tls\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:21.716834 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.716146    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a85cca66-a438-4117-9f50-977c7f528b7c-nginx-conf\") pod \"networking-console-plugin-86c478bb88-lpg7w\" (UID: \"a85cca66-a438-4117-9f50-977c7f528b7c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lpg7w"
Oct 30 19:16:21.716834 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.716160    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9f302411-01be-438c-a30c-70c944afd630-registry-certificates\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:21.716834 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.716305    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2d261662-727d-4d68-9f4a-9790ecf16d46-konnectivity-ca\") pod \"konnectivity-agent-lqmqr\" (UID: \"2d261662-727d-4d68-9f4a-9790ecf16d46\") " pod="kube-system/konnectivity-agent-lqmqr"
Oct 30 19:16:21.716834 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.716386    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh979\" (UniqueName: \"kubernetes.io/projected/06397fd4-3377-4eaa-8879-ebbb5943fbda-kube-api-access-zh979\") pod \"cluster-monitoring-operator-884576c57-2sdkw\" (UID: \"06397fd4-3377-4eaa-8879-ebbb5943fbda\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2sdkw"
Oct 30 19:16:21.716834 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.716470    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-run-systemd\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.716834 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.716516    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a64211d0-1c48-41d2-b254-4ae84d5097d7-installation-pull-secrets\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:21.716834 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.716546    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f1ea27b2-80df-404c-ac39-460f495957b2-serviceca\") pod \"node-ca-zj8d6\" (UID: \"f1ea27b2-80df-404c-ac39-460f495957b2\") " pod="openshift-image-registry/node-ca-zj8d6"
Oct 30 19:16:21.716834 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.716589    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4dbf226e-336f-42c6-a766-cc0ddfe68bef-cni-binary-copy\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:16:21.716834 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.716607    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2d261662-727d-4d68-9f4a-9790ecf16d46-konnectivity-ca\") pod \"konnectivity-agent-lqmqr\" (UID: \"2d261662-727d-4d68-9f4a-9790ecf16d46\") " pod="kube-system/konnectivity-agent-lqmqr"
Oct 30 19:16:21.716834 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.716616    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/353ebfb4-7e11-41de-b992-277443f56960-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-62bhw\" (UID: \"353ebfb4-7e11-41de-b992-277443f56960\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-62bhw"
Oct 30 19:16:21.716834 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.716640    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28sgd\" (UniqueName: \"kubernetes.io/projected/247ff425-bed3-4297-b5c0-cc32b21bc179-kube-api-access-28sgd\") pod \"router-default-75866dc98c-4mnhf\" (UID: \"247ff425-bed3-4297-b5c0-cc32b21bc179\") " pod="openshift-ingress/router-default-75866dc98c-4mnhf"
Oct 30 19:16:21.716834 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.716696    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-sys\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:16:21.716834 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.716759    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a64211d0-1c48-41d2-b254-4ae84d5097d7-installation-pull-secrets\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:21.716834 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.716776    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8e297a2-a78a-4665-86d0-6d842bbcaa51-metrics-certs\") pod \"network-metrics-daemon-n9cws\" (UID: \"e8e297a2-a78a-4665-86d0-6d842bbcaa51\") " pod="openshift-multus/network-metrics-daemon-n9cws"
Oct 30 19:16:21.716834 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.716802    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6chk\" (UniqueName: \"kubernetes.io/projected/76e21916-0744-4499-a0cb-64a7a44a8285-kube-api-access-v6chk\") pod \"console-operator-7cd56b8dbc-cmw6w\" (UID: \"76e21916-0744-4499-a0cb-64a7a44a8285\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cmw6w"
Oct 30 19:16:21.716834 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.716808    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/353ebfb4-7e11-41de-b992-277443f56960-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-62bhw\" (UID: \"353ebfb4-7e11-41de-b992-277443f56960\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-62bhw"
Oct 30 19:16:21.716834 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.716839    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5338b461-6b7c-474b-8458-d1b53775ee0f-trusted-ca-bundle\") pod \"periodic-gathering-2lcwk-gjv8f\" (UID: \"5338b461-6b7c-474b-8458-d1b53775ee0f\") " pod="openshift-insights/periodic-gathering-2lcwk-gjv8f"
Oct 30 19:16:21.717755 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.716873    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-etc-sysconfig\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:16:21.717755 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.716899    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4dbf226e-336f-42c6-a766-cc0ddfe68bef-cni-binary-copy\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:16:21.717755 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.716909    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8ww9\" (UniqueName: \"kubernetes.io/projected/00b97df0-857e-4ad9-b33a-4a59777b6f62-kube-api-access-w8ww9\") pod \"kas-connection-checker-685895c7c9-6b8sd\" (UID: \"00b97df0-857e-4ad9-b33a-4a59777b6f62\") " pod="kube-system/kas-connection-checker-685895c7c9-6b8sd"
Oct 30 19:16:21.717755 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.716930    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9f302411-01be-438c-a30c-70c944afd630-registry-certificates\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:21.717755 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.716939    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aeef13dc-e06e-49c5-b893-64c7f73a7e44-data-volume\") pod \"insights-runtime-extractor-smgf6\" (UID: \"aeef13dc-e06e-49c5-b893-64c7f73a7e44\") " pod="openshift-insights/insights-runtime-extractor-smgf6"
Oct 30 19:16:21.717755 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.716983    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4dbf226e-336f-42c6-a766-cc0ddfe68bef-multus-daemon-config\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:16:21.717755 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.717032    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-host-run-netns\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.717755 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.717044    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aeef13dc-e06e-49c5-b893-64c7f73a7e44-data-volume\") pod \"insights-runtime-extractor-smgf6\" (UID: \"aeef13dc-e06e-49c5-b893-64c7f73a7e44\") " pod="openshift-insights/insights-runtime-extractor-smgf6"
Oct 30 19:16:21.717755 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.717060    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0269ffa6-ce06-43f3-be2a-58f6aa3e6964-tmp-dir\") pod \"dns-default-gkn6q\" (UID: \"0269ffa6-ce06-43f3-be2a-58f6aa3e6964\") " pod="openshift-dns/dns-default-gkn6q"
Oct 30 19:16:21.717755 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.717199    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4dbf226e-336f-42c6-a766-cc0ddfe68bef-multus-daemon-config\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:16:21.717755 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.717202    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndw2r\" (UniqueName: \"kubernetes.io/projected/9201f6f2-73b9-45cd-9950-3a07914d8ea9-kube-api-access-ndw2r\") pod \"kas-connection-checker-685895c7c9-65kc2\" (UID: \"9201f6f2-73b9-45cd-9950-3a07914d8ea9\") " pod="kube-system/kas-connection-checker-685895c7c9-65kc2"
Oct 30 19:16:21.717755 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.717228    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5338b461-6b7c-474b-8458-d1b53775ee0f-trusted-ca-bundle\") pod \"periodic-gathering-2lcwk-gjv8f\" (UID: \"5338b461-6b7c-474b-8458-d1b53775ee0f\") " pod="openshift-insights/periodic-gathering-2lcwk-gjv8f"
Oct 30 19:16:21.717755 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.717265    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0269ffa6-ce06-43f3-be2a-58f6aa3e6964-tmp-dir\") pod \"dns-default-gkn6q\" (UID: \"0269ffa6-ce06-43f3-be2a-58f6aa3e6964\") " pod="openshift-dns/dns-default-gkn6q"
Oct 30 19:16:21.717755 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.717272    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfglw\" (UniqueName: \"kubernetes.io/projected/2d52a94d-72ac-4e5c-9472-104399a21a3b-kube-api-access-zfglw\") pod \"network-check-source-6f78d59f99-bvcb4\" (UID: \"2d52a94d-72ac-4e5c-9472-104399a21a3b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bvcb4"
Oct 30 19:16:21.717755 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.717275    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f1ea27b2-80df-404c-ac39-460f495957b2-serviceca\") pod \"node-ca-zj8d6\" (UID: \"f1ea27b2-80df-404c-ac39-460f495957b2\") " pod="openshift-image-registry/node-ca-zj8d6"
Oct 30 19:16:21.717755 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.717403    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-log-socket\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.717755 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.717435    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gnlf\" (UniqueName: \"kubernetes.io/projected/47e9832e-bc96-445a-a93e-9d24f84db3b3-kube-api-access-6gnlf\") pod \"network-check-target-n9zx2\" (UID: \"47e9832e-bc96-445a-a93e-9d24f84db3b3\") " pod="openshift-network-diagnostics/network-check-target-n9zx2"
Oct 30 19:16:21.717755 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.717495    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a37bca55-21cf-4b76-9c05-e2c951647bf8-signing-cabundle\") pod \"service-ca-5fffc97bdf-b8cs2\" (UID: \"a37bca55-21cf-4b76-9c05-e2c951647bf8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-b8cs2"
Oct 30 19:16:21.717755 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.717567    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dda4158a-327b-4b73-8673-3df1c1c0e400-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w2262\" (UID: \"dda4158a-327b-4b73-8673-3df1c1c0e400\") " pod="openshift-multus/multus-additional-cni-plugins-w2262"
Oct 30 19:16:21.717755 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.717595    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/247ff425-bed3-4297-b5c0-cc32b21bc179-default-certificate\") pod \"router-default-75866dc98c-4mnhf\" (UID: \"247ff425-bed3-4297-b5c0-cc32b21bc179\") " pod="openshift-ingress/router-default-75866dc98c-4mnhf"
Oct 30 19:16:21.717755 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.717683    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6331f51e-cf5e-4ba2-9eda-35fd62b14091-host-slash\") pod \"iptables-alerter-26xq4\" (UID: \"6331f51e-cf5e-4ba2-9eda-35fd62b14091\") " pod="openshift-network-operator/iptables-alerter-26xq4"
Oct 30 19:16:21.717755 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.717718    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6ecb643b-1dce-4523-89df-93b22ad8ce6f-dbus\") pod \"global-pull-secret-syncer-wg2df\" (UID: \"6ecb643b-1dce-4523-89df-93b22ad8ce6f\") " pod="kube-system/global-pull-secret-syncer-wg2df"
Oct 30 19:16:21.717755 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.717731    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a37bca55-21cf-4b76-9c05-e2c951647bf8-signing-cabundle\") pod \"service-ca-5fffc97bdf-b8cs2\" (UID: \"a37bca55-21cf-4b76-9c05-e2c951647bf8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-b8cs2"
Oct 30 19:16:21.717755 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.717765    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a64211d0-1c48-41d2-b254-4ae84d5097d7-ca-trust-extracted\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:21.718766 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.717784    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/247ff425-bed3-4297-b5c0-cc32b21bc179-default-certificate\") pod \"router-default-75866dc98c-4mnhf\" (UID: \"247ff425-bed3-4297-b5c0-cc32b21bc179\") " pod="openshift-ingress/router-default-75866dc98c-4mnhf"
Oct 30 19:16:21.718766 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.717793    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-multus-cni-dir\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:16:21.718766 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.717802    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dda4158a-327b-4b73-8673-3df1c1c0e400-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w2262\" (UID: \"dda4158a-327b-4b73-8673-3df1c1c0e400\") " pod="openshift-multus/multus-additional-cni-plugins-w2262"
Oct 30 19:16:21.718766 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.717846    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dc581227-9542-4571-aced-63d191692a41-env-overrides\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.718766 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.717869    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a64211d0-1c48-41d2-b254-4ae84d5097d7-ca-trust-extracted\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:21.718766 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.717885    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-bound-sa-token\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:21.718766 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.717975    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ecb643b-1dce-4523-89df-93b22ad8ce6f-original-pull-secret\") pod \"global-pull-secret-syncer-wg2df\" (UID: \"6ecb643b-1dce-4523-89df-93b22ad8ce6f\") " pod="kube-system/global-pull-secret-syncer-wg2df"
Oct 30 19:16:21.718766 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.718005    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/dc581227-9542-4571-aced-63d191692a41-env-overrides\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.718766 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.718087    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/06397fd4-3377-4eaa-8879-ebbb5943fbda-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-2sdkw\" (UID: \"06397fd4-3377-4eaa-8879-ebbb5943fbda\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2sdkw"
Oct 30 19:16:21.718766 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.718134    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/247ff425-bed3-4297-b5c0-cc32b21bc179-stats-auth\") pod \"router-default-75866dc98c-4mnhf\" (UID: \"247ff425-bed3-4297-b5c0-cc32b21bc179\") " pod="openshift-ingress/router-default-75866dc98c-4mnhf"
Oct 30 19:16:21.718766 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.718165    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-host-slash\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.718766 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.718186    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dc581227-9542-4571-aced-63d191692a41-ovn-node-metrics-cert\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.718766 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.718208    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6ecb643b-1dce-4523-89df-93b22ad8ce6f-kubelet-config\") pod \"global-pull-secret-syncer-wg2df\" (UID: \"6ecb643b-1dce-4523-89df-93b22ad8ce6f\") " pod="kube-system/global-pull-secret-syncer-wg2df"
Oct 30 19:16:21.718766 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.718226    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-host-run-netns\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:16:21.718766 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.718244    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a64211d0-1c48-41d2-b254-4ae84d5097d7-registry-certificates\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:21.718766 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.718253    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ecb643b-1dce-4523-89df-93b22ad8ce6f-original-pull-secret\") pod \"global-pull-secret-syncer-wg2df\" (UID: \"6ecb643b-1dce-4523-89df-93b22ad8ce6f\") " pod="kube-system/global-pull-secret-syncer-wg2df"
Oct 30 19:16:21.718766 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.718264    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x795\" (UniqueName: \"kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-kube-api-access-6x795\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:21.718766 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.718322    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/76e21916-0744-4499-a0cb-64a7a44a8285-serving-cert\") pod \"console-operator-7cd56b8dbc-cmw6w\" (UID: \"76e21916-0744-4499-a0cb-64a7a44a8285\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cmw6w"
Oct 30 19:16:21.718766 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.718383    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/247ff425-bed3-4297-b5c0-cc32b21bc179-stats-auth\") pod \"router-default-75866dc98c-4mnhf\" (UID: \"247ff425-bed3-4297-b5c0-cc32b21bc179\") " pod="openshift-ingress/router-default-75866dc98c-4mnhf"
Oct 30 19:16:21.718766 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.718433    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-system-cni-dir\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:16:21.718766 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.718554    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/76e21916-0744-4499-a0cb-64a7a44a8285-serving-cert\") pod \"console-operator-7cd56b8dbc-cmw6w\" (UID: \"76e21916-0744-4499-a0cb-64a7a44a8285\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cmw6w"
Oct 30 19:16:21.718766 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.718566    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e5002691-8257-4dab-a88b-3c1b60e8e6f8-serving-cert\") pod \"insights-operator-7c4986d449-fz9hf\" (UID: \"e5002691-8257-4dab-a88b-3c1b60e8e6f8\") " pod="openshift-insights/insights-operator-7c4986d449-fz9hf"
Oct 30 19:16:21.718766 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.718605    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5338b461-6b7c-474b-8458-d1b53775ee0f-service-ca-bundle\") pod \"periodic-gathering-2lcwk-gjv8f\" (UID: \"5338b461-6b7c-474b-8458-d1b53775ee0f\") " pod="openshift-insights/periodic-gathering-2lcwk-gjv8f"
Oct 30 19:16:21.718766 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.718656    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9f302411-01be-438c-a30c-70c944afd630-image-registry-private-configuration\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:21.718766 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.718694    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5ldg\" (UniqueName: \"kubernetes.io/projected/6331f51e-cf5e-4ba2-9eda-35fd62b14091-kube-api-access-q5ldg\") pod \"iptables-alerter-26xq4\" (UID: \"6331f51e-cf5e-4ba2-9eda-35fd62b14091\") " pod="openshift-network-operator/iptables-alerter-26xq4"
Oct 30 19:16:21.718766 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.718721    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tngcd\" (UniqueName: \"kubernetes.io/projected/f1ea27b2-80df-404c-ac39-460f495957b2-kube-api-access-tngcd\") pod \"node-ca-zj8d6\" (UID: \"f1ea27b2-80df-404c-ac39-460f495957b2\") " pod="openshift-image-registry/node-ca-zj8d6"
Oct 30 19:16:21.720161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.718812    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/dc581227-9542-4571-aced-63d191692a41-ovn-node-metrics-cert\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.720161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.718818    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-host-cni-netd\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.720161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.718818    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/06397fd4-3377-4eaa-8879-ebbb5943fbda-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-2sdkw\" (UID: \"06397fd4-3377-4eaa-8879-ebbb5943fbda\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2sdkw"
Oct 30 19:16:21.720161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.718849    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xrrr\" (UniqueName: \"kubernetes.io/projected/5338b461-6b7c-474b-8458-d1b53775ee0f-kube-api-access-4xrrr\") pod \"periodic-gathering-2lcwk-gjv8f\" (UID: \"5338b461-6b7c-474b-8458-d1b53775ee0f\") " pod="openshift-insights/periodic-gathering-2lcwk-gjv8f"
Oct 30 19:16:21.720161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.718879    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/cbc4f7c7-bebd-4200-833e-1afbf3c2c81a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-p5nzf\" (UID: \"cbc4f7c7-bebd-4200-833e-1afbf3c2c81a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p5nzf"
Oct 30 19:16:21.720161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.718912    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pfsl\" (UniqueName: \"kubernetes.io/projected/353ebfb4-7e11-41de-b992-277443f56960-kube-api-access-2pfsl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-62bhw\" (UID: \"353ebfb4-7e11-41de-b992-277443f56960\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-62bhw"
Oct 30 19:16:21.720161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.718937    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9f302411-01be-438c-a30c-70c944afd630-image-registry-private-configuration\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:21.720161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.718882    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5338b461-6b7c-474b-8458-d1b53775ee0f-service-ca-bundle\") pod \"periodic-gathering-2lcwk-gjv8f\" (UID: \"5338b461-6b7c-474b-8458-d1b53775ee0f\") " pod="openshift-insights/periodic-gathering-2lcwk-gjv8f"
Oct 30 19:16:21.720161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.718943    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/76e21916-0744-4499-a0cb-64a7a44a8285-config\") pod \"console-operator-7cd56b8dbc-cmw6w\" (UID: \"76e21916-0744-4499-a0cb-64a7a44a8285\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cmw6w"
Oct 30 19:16:21.720161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.718999    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-etc-modprobe-d\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:16:21.720161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.719027    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-registry-tls\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:21.720161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.719055    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tnx8\" (UniqueName: \"kubernetes.io/projected/c243f1d8-1c43-4eb8-93ec-cb28d72cc8a1-kube-api-access-4tnx8\") pod \"volume-data-source-validator-58c9f5f46d-mg98j\" (UID: \"c243f1d8-1c43-4eb8-93ec-cb28d72cc8a1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-mg98j"
Oct 30 19:16:21.720161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.719116    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e5002691-8257-4dab-a88b-3c1b60e8e6f8-serving-cert\") pod \"insights-operator-7c4986d449-fz9hf\" (UID: \"e5002691-8257-4dab-a88b-3c1b60e8e6f8\") " pod="openshift-insights/insights-operator-7c4986d449-fz9hf"
Oct 30 19:16:21.720161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.719141    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-host-run-ovn-kubernetes\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.720161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.719223    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/76e21916-0744-4499-a0cb-64a7a44a8285-config\") pod \"console-operator-7cd56b8dbc-cmw6w\" (UID: \"76e21916-0744-4499-a0cb-64a7a44a8285\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cmw6w"
Oct 30 19:16:21.720161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.719233    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a64211d0-1c48-41d2-b254-4ae84d5097d7-image-registry-private-configuration\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:21.720161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.719352    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/765d8d9d-b89f-419f-b237-3f8034db1bed-cert\") pod \"ingress-canary-9jkcb\" (UID: \"765d8d9d-b89f-419f-b237-3f8034db1bed\") " pod="openshift-ingress-canary/ingress-canary-9jkcb"
Oct 30 19:16:21.720161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.719413    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-host-var-lib-cni-multus\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:16:21.720161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.719467    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dc581227-9542-4571-aced-63d191692a41-ovnkube-config\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.720161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.719473    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/cbc4f7c7-bebd-4200-833e-1afbf3c2c81a-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-p5nzf\" (UID: \"cbc4f7c7-bebd-4200-833e-1afbf3c2c81a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p5nzf"
Oct 30 19:16:21.720161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.719484    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a64211d0-1c48-41d2-b254-4ae84d5097d7-image-registry-private-configuration\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:21.720161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.719497    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-bound-sa-token\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:21.720161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.719534    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a64211d0-1c48-41d2-b254-4ae84d5097d7-registry-certificates\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:21.720161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.719572    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lksw9\" (UniqueName: \"kubernetes.io/projected/36ab52a2-65e2-4af6-a872-9052e12714d1-kube-api-access-lksw9\") pod \"node-resolver-xjjqh\" (UID: \"36ab52a2-65e2-4af6-a872-9052e12714d1\") " pod="openshift-dns/node-resolver-xjjqh"
Oct 30 19:16:21.720161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.719602    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dda4158a-327b-4b73-8673-3df1c1c0e400-cni-binary-copy\") pod \"multus-additional-cni-plugins-w2262\" (UID: \"dda4158a-327b-4b73-8673-3df1c1c0e400\") " pod="openshift-multus/multus-additional-cni-plugins-w2262"
Oct 30 19:16:21.720161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.719632    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/765d8d9d-b89f-419f-b237-3f8034db1bed-cert\") pod \"ingress-canary-9jkcb\" (UID: \"765d8d9d-b89f-419f-b237-3f8034db1bed\") " pod="openshift-ingress-canary/ingress-canary-9jkcb"
Oct 30 19:16:21.720161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.719638    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/06397fd4-3377-4eaa-8879-ebbb5943fbda-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2sdkw\" (UID: \"06397fd4-3377-4eaa-8879-ebbb5943fbda\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2sdkw"
Oct 30 19:16:21.720161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.719677    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-host-run-multus-certs\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:16:21.720161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.719707    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.720161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.719708    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/dc581227-9542-4571-aced-63d191692a41-ovnkube-config\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.720161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.719762    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f1ea27b2-80df-404c-ac39-460f495957b2-host\") pod \"node-ca-zj8d6\" (UID: \"f1ea27b2-80df-404c-ac39-460f495957b2\") " pod="openshift-image-registry/node-ca-zj8d6"
Oct 30 19:16:21.720161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.719879    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dda4158a-327b-4b73-8673-3df1c1c0e400-cni-binary-copy\") pod \"multus-additional-cni-plugins-w2262\" (UID: \"dda4158a-327b-4b73-8673-3df1c1c0e400\") " pod="openshift-multus/multus-additional-cni-plugins-w2262"
Oct 30 19:16:21.720161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.719885    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njpgh\" (UniqueName: \"kubernetes.io/projected/765d8d9d-b89f-419f-b237-3f8034db1bed-kube-api-access-njpgh\") pod \"ingress-canary-9jkcb\" (UID: \"765d8d9d-b89f-419f-b237-3f8034db1bed\") " pod="openshift-ingress-canary/ingress-canary-9jkcb"
Oct 30 19:16:21.720161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.719994    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-host-run-k8s-cni-cncf-io\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:16:21.720161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.720028    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-systemd-units\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.720161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.720046    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9f302411-01be-438c-a30c-70c944afd630-trusted-ca\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:21.720161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.720079    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/68201777-2ec5-43b2-b587-81e585865d08-config\") pod \"service-ca-operator-544d749c8c-vd8tj\" (UID: \"68201777-2ec5-43b2-b587-81e585865d08\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vd8tj"
Oct 30 19:16:21.720161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.720095    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5c96\" (UniqueName: \"kubernetes.io/projected/dda4158a-327b-4b73-8673-3df1c1c0e400-kube-api-access-m5c96\") pod \"multus-additional-cni-plugins-w2262\" (UID: \"dda4158a-327b-4b73-8673-3df1c1c0e400\") " pod="openshift-multus/multus-additional-cni-plugins-w2262"
Oct 30 19:16:21.720161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.720191    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsjcc\" (UniqueName: \"kubernetes.io/projected/c84a5c22-7a46-4de1-aef7-32cb9a5453bb-kube-api-access-zsjcc\") pod \"kas-connection-checker-685895c7c9-zqvrw\" (UID: \"c84a5c22-7a46-4de1-aef7-32cb9a5453bb\") " pod="kube-system/kas-connection-checker-685895c7c9-zqvrw"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.720212    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8ptq\" (UniqueName: \"kubernetes.io/projected/77fbf490-7af3-4bde-8222-4038e2219fff-kube-api-access-l8ptq\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.720229    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9f302411-01be-438c-a30c-70c944afd630-installation-pull-secrets\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.720244    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68201777-2ec5-43b2-b587-81e585865d08-serving-cert\") pod \"service-ca-operator-544d749c8c-vd8tj\" (UID: \"68201777-2ec5-43b2-b587-81e585865d08\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vd8tj"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.720284    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s56kf\" (UniqueName: \"kubernetes.io/projected/68201777-2ec5-43b2-b587-81e585865d08-kube-api-access-s56kf\") pod \"service-ca-operator-544d749c8c-vd8tj\" (UID: \"68201777-2ec5-43b2-b587-81e585865d08\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vd8tj"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.720302    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dda4158a-327b-4b73-8673-3df1c1c0e400-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w2262\" (UID: \"dda4158a-327b-4b73-8673-3df1c1c0e400\") " pod="openshift-multus/multus-additional-cni-plugins-w2262"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.720323    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-etc-openvswitch\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.720339    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5002691-8257-4dab-a88b-3c1b60e8e6f8-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-fz9hf\" (UID: \"e5002691-8257-4dab-a88b-3c1b60e8e6f8\") " pod="openshift-insights/insights-operator-7c4986d449-fz9hf"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.720355    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6bsd\" (UniqueName: \"kubernetes.io/projected/0269ffa6-ce06-43f3-be2a-58f6aa3e6964-kube-api-access-b6bsd\") pod \"dns-default-gkn6q\" (UID: \"0269ffa6-ce06-43f3-be2a-58f6aa3e6964\") " pod="openshift-dns/dns-default-gkn6q"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.720369    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-host-cni-bin\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.720385    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a37bca55-21cf-4b76-9c05-e2c951647bf8-signing-key\") pod \"service-ca-5fffc97bdf-b8cs2\" (UID: \"a37bca55-21cf-4b76-9c05-e2c951647bf8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-b8cs2"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.720420    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/68201777-2ec5-43b2-b587-81e585865d08-config\") pod \"service-ca-operator-544d749c8c-vd8tj\" (UID: \"68201777-2ec5-43b2-b587-81e585865d08\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vd8tj"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.720427    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9f302411-01be-438c-a30c-70c944afd630-trusted-ca\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.720471    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/128a8dad-66f8-414b-8c1d-14c7d3b58d25-registration-dir\") pod \"aws-ebs-csi-driver-node-6k5kr\" (UID: \"128a8dad-66f8-414b-8c1d-14c7d3b58d25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.720504    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77fbf490-7af3-4bde-8222-4038e2219fff-tmp\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.720523    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6331f51e-cf5e-4ba2-9eda-35fd62b14091-iptables-alerter-script\") pod \"iptables-alerter-26xq4\" (UID: \"6331f51e-cf5e-4ba2-9eda-35fd62b14091\") " pod="openshift-network-operator/iptables-alerter-26xq4"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.720539    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/76e21916-0744-4499-a0cb-64a7a44a8285-trusted-ca\") pod \"console-operator-7cd56b8dbc-cmw6w\" (UID: \"76e21916-0744-4499-a0cb-64a7a44a8285\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cmw6w"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.720555    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxkzk\" (UniqueName: \"kubernetes.io/projected/128a8dad-66f8-414b-8c1d-14c7d3b58d25-kube-api-access-jxkzk\") pod \"aws-ebs-csi-driver-node-6k5kr\" (UID: \"128a8dad-66f8-414b-8c1d-14c7d3b58d25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.720570    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-hostroot\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.720585    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/128a8dad-66f8-414b-8c1d-14c7d3b58d25-socket-dir\") pod \"aws-ebs-csi-driver-node-6k5kr\" (UID: \"128a8dad-66f8-414b-8c1d-14c7d3b58d25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.720601    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-lib-modules\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.720616    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dda4158a-327b-4b73-8673-3df1c1c0e400-cnibin\") pod \"multus-additional-cni-plugins-w2262\" (UID: \"dda4158a-327b-4b73-8673-3df1c1c0e400\") " pod="openshift-multus/multus-additional-cni-plugins-w2262"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.720644    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a37bca55-21cf-4b76-9c05-e2c951647bf8-signing-key\") pod \"service-ca-5fffc97bdf-b8cs2\" (UID: \"a37bca55-21cf-4b76-9c05-e2c951647bf8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-b8cs2"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.720651    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e5002691-8257-4dab-a88b-3c1b60e8e6f8-snapshots\") pod \"insights-operator-7c4986d449-fz9hf\" (UID: \"e5002691-8257-4dab-a88b-3c1b60e8e6f8\") " pod="openshift-insights/insights-operator-7c4986d449-fz9hf"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.720669    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rz2d\" (UniqueName: \"kubernetes.io/projected/e5002691-8257-4dab-a88b-3c1b60e8e6f8-kube-api-access-9rz2d\") pod \"insights-operator-7c4986d449-fz9hf\" (UID: \"e5002691-8257-4dab-a88b-3c1b60e8e6f8\") " pod="openshift-insights/insights-operator-7c4986d449-fz9hf"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.720684    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0269ffa6-ce06-43f3-be2a-58f6aa3e6964-metrics-tls\") pod \"dns-default-gkn6q\" (UID: \"0269ffa6-ce06-43f3-be2a-58f6aa3e6964\") " pod="openshift-dns/dns-default-gkn6q"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.720701    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vk66\" (UniqueName: \"kubernetes.io/projected/a37bca55-21cf-4b76-9c05-e2c951647bf8-kube-api-access-2vk66\") pod \"service-ca-5fffc97bdf-b8cs2\" (UID: \"a37bca55-21cf-4b76-9c05-e2c951647bf8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-b8cs2"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.720716    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-run\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.720748    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpb5t\" (UniqueName: \"kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-kube-api-access-hpb5t\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.720774    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2d261662-727d-4d68-9f4a-9790ecf16d46-agent-certs\") pod \"konnectivity-agent-lqmqr\" (UID: \"2d261662-727d-4d68-9f4a-9790ecf16d46\") " pod="kube-system/konnectivity-agent-lqmqr"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.720810    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aeef13dc-e06e-49c5-b893-64c7f73a7e44-crio-socket\") pod \"insights-runtime-extractor-smgf6\" (UID: \"aeef13dc-e06e-49c5-b893-64c7f73a7e44\") " pod="openshift-insights/insights-runtime-extractor-smgf6"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.720890    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77fbf490-7af3-4bde-8222-4038e2219fff-tmp\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.720972    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5338b461-6b7c-474b-8458-d1b53775ee0f-archives-path\") pod \"periodic-gathering-2lcwk-gjv8f\" (UID: \"5338b461-6b7c-474b-8458-d1b53775ee0f\") " pod="openshift-insights/periodic-gathering-2lcwk-gjv8f"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.721002    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-var-lib-kubelet\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.721026    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aeef13dc-e06e-49c5-b893-64c7f73a7e44-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-smgf6\" (UID: \"aeef13dc-e06e-49c5-b893-64c7f73a7e44\") " pod="openshift-insights/insights-runtime-extractor-smgf6"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.721053    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aeef13dc-e06e-49c5-b893-64c7f73a7e44-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-smgf6\" (UID: \"aeef13dc-e06e-49c5-b893-64c7f73a7e44\") " pod="openshift-insights/insights-runtime-extractor-smgf6"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.721068    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9f302411-01be-438c-a30c-70c944afd630-installation-pull-secrets\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.721098    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2d261662-727d-4d68-9f4a-9790ecf16d46-agent-certs\") pod \"konnectivity-agent-lqmqr\" (UID: \"2d261662-727d-4d68-9f4a-9790ecf16d46\") " pod="kube-system/konnectivity-agent-lqmqr"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.721127    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/76e21916-0744-4499-a0cb-64a7a44a8285-trusted-ca\") pod \"console-operator-7cd56b8dbc-cmw6w\" (UID: \"76e21916-0744-4499-a0cb-64a7a44a8285\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cmw6w"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.721139    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6331f51e-cf5e-4ba2-9eda-35fd62b14091-iptables-alerter-script\") pod \"iptables-alerter-26xq4\" (UID: \"6331f51e-cf5e-4ba2-9eda-35fd62b14091\") " pod="openshift-network-operator/iptables-alerter-26xq4"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.721169    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e5002691-8257-4dab-a88b-3c1b60e8e6f8-snapshots\") pod \"insights-operator-7c4986d449-fz9hf\" (UID: \"e5002691-8257-4dab-a88b-3c1b60e8e6f8\") " pod="openshift-insights/insights-operator-7c4986d449-fz9hf"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.721170    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-cnibin\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.721183    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5338b461-6b7c-474b-8458-d1b53775ee0f-archives-path\") pod \"periodic-gathering-2lcwk-gjv8f\" (UID: \"5338b461-6b7c-474b-8458-d1b53775ee0f\") " pod="openshift-insights/periodic-gathering-2lcwk-gjv8f"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.721228    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-etc-kubernetes\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.721266    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbndd\" (UniqueName: \"kubernetes.io/projected/4dbf226e-336f-42c6-a766-cc0ddfe68bef-kube-api-access-lbndd\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.721347    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5002691-8257-4dab-a88b-3c1b60e8e6f8-service-ca-bundle\") pod \"insights-operator-7c4986d449-fz9hf\" (UID: \"e5002691-8257-4dab-a88b-3c1b60e8e6f8\") " pod="openshift-insights/insights-operator-7c4986d449-fz9hf"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.721390    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/128a8dad-66f8-414b-8c1d-14c7d3b58d25-device-dir\") pod \"aws-ebs-csi-driver-node-6k5kr\" (UID: \"128a8dad-66f8-414b-8c1d-14c7d3b58d25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.721447    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68201777-2ec5-43b2-b587-81e585865d08-serving-cert\") pod \"service-ca-operator-544d749c8c-vd8tj\" (UID: \"68201777-2ec5-43b2-b587-81e585865d08\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vd8tj"
Oct 30 19:16:21.721840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.721863    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5002691-8257-4dab-a88b-3c1b60e8e6f8-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-fz9hf\" (UID: \"e5002691-8257-4dab-a88b-3c1b60e8e6f8\") " pod="openshift-insights/insights-operator-7c4986d449-fz9hf"
Oct 30 19:16:21.723522 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.721876    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5002691-8257-4dab-a88b-3c1b60e8e6f8-service-ca-bundle\") pod \"insights-operator-7c4986d449-fz9hf\" (UID: \"e5002691-8257-4dab-a88b-3c1b60e8e6f8\") " pod="openshift-insights/insights-operator-7c4986d449-fz9hf"
Oct 30 19:16:21.723797 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.723779    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr447\" (UniqueName: \"kubernetes.io/projected/16705cc4-1ecc-47d1-9f4a-bb7eadc6294c-kube-api-access-qr447\") pod \"migrator-66df5d496b-vv4df\" (UID: \"16705cc4-1ecc-47d1-9f4a-bb7eadc6294c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vv4df"
Oct 30 19:16:21.723867 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.723855    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq4kp\" (UniqueName: \"kubernetes.io/projected/aeef13dc-e06e-49c5-b893-64c7f73a7e44-kube-api-access-tq4kp\") pod \"insights-runtime-extractor-smgf6\" (UID: \"aeef13dc-e06e-49c5-b893-64c7f73a7e44\") " pod="openshift-insights/insights-runtime-extractor-smgf6"
Oct 30 19:16:21.724063 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.724046    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcgjp\" (UniqueName: \"kubernetes.io/projected/cbc4f7c7-bebd-4200-833e-1afbf3c2c81a-kube-api-access-mcgjp\") pod \"cluster-samples-operator-6ddb6c8b94-p5nzf\" (UID: \"cbc4f7c7-bebd-4200-833e-1afbf3c2c81a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p5nzf"
Oct 30 19:16:21.724174 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.724154    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh979\" (UniqueName: \"kubernetes.io/projected/06397fd4-3377-4eaa-8879-ebbb5943fbda-kube-api-access-zh979\") pod \"cluster-monitoring-operator-884576c57-2sdkw\" (UID: \"06397fd4-3377-4eaa-8879-ebbb5943fbda\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2sdkw"
Oct 30 19:16:21.724428 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.724409    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hx4s\" (UniqueName: \"kubernetes.io/projected/853a64ec-147d-42eb-8946-e335de7878d3-kube-api-access-7hx4s\") pod \"migrator-66df5d496b-qggf8\" (UID: \"853a64ec-147d-42eb-8946-e335de7878d3\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qggf8"
Oct 30 19:16:21.724847 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.724826    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xfxj\" (UniqueName: \"kubernetes.io/projected/e8e297a2-a78a-4665-86d0-6d842bbcaa51-kube-api-access-5xfxj\") pod \"network-metrics-daemon-n9cws\" (UID: \"e8e297a2-a78a-4665-86d0-6d842bbcaa51\") " pod="openshift-multus/network-metrics-daemon-n9cws"
Oct 30 19:16:21.724977 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.724948    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m9lt\" (UniqueName: \"kubernetes.io/projected/dc581227-9542-4571-aced-63d191692a41-kube-api-access-4m9lt\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.725890 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.725861    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6chk\" (UniqueName: \"kubernetes.io/projected/76e21916-0744-4499-a0cb-64a7a44a8285-kube-api-access-v6chk\") pod \"console-operator-7cd56b8dbc-cmw6w\" (UID: \"76e21916-0744-4499-a0cb-64a7a44a8285\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cmw6w"
Oct 30 19:16:21.725961 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.725903    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-bound-sa-token\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:21.726154 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.726136    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8ww9\" (UniqueName: \"kubernetes.io/projected/00b97df0-857e-4ad9-b33a-4a59777b6f62-kube-api-access-w8ww9\") pod \"kas-connection-checker-685895c7c9-6b8sd\" (UID: \"00b97df0-857e-4ad9-b33a-4a59777b6f62\") " pod="kube-system/kas-connection-checker-685895c7c9-6b8sd"
Oct 30 19:16:21.726485 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.726461    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gnlf\" (UniqueName: \"kubernetes.io/projected/47e9832e-bc96-445a-a93e-9d24f84db3b3-kube-api-access-6gnlf\") pod \"network-check-target-n9zx2\" (UID: \"47e9832e-bc96-445a-a93e-9d24f84db3b3\") " pod="openshift-network-diagnostics/network-check-target-n9zx2"
Oct 30 19:16:21.726554 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.726468    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28sgd\" (UniqueName: \"kubernetes.io/projected/247ff425-bed3-4297-b5c0-cc32b21bc179-kube-api-access-28sgd\") pod \"router-default-75866dc98c-4mnhf\" (UID: \"247ff425-bed3-4297-b5c0-cc32b21bc179\") " pod="openshift-ingress/router-default-75866dc98c-4mnhf"
Oct 30 19:16:21.726612 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.726594    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfglw\" (UniqueName: \"kubernetes.io/projected/2d52a94d-72ac-4e5c-9472-104399a21a3b-kube-api-access-zfglw\") pod \"network-check-source-6f78d59f99-bvcb4\" (UID: \"2d52a94d-72ac-4e5c-9472-104399a21a3b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bvcb4"
Oct 30 19:16:21.727112 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.727089    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndw2r\" (UniqueName: \"kubernetes.io/projected/9201f6f2-73b9-45cd-9950-3a07914d8ea9-kube-api-access-ndw2r\") pod \"kas-connection-checker-685895c7c9-65kc2\" (UID: \"9201f6f2-73b9-45cd-9950-3a07914d8ea9\") " pod="kube-system/kas-connection-checker-685895c7c9-65kc2"
Oct 30 19:16:21.728047 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.728031    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5ldg\" (UniqueName: \"kubernetes.io/projected/6331f51e-cf5e-4ba2-9eda-35fd62b14091-kube-api-access-q5ldg\") pod \"iptables-alerter-26xq4\" (UID: \"6331f51e-cf5e-4ba2-9eda-35fd62b14091\") " pod="openshift-network-operator/iptables-alerter-26xq4"
Oct 30 19:16:21.728152 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.728137    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x795\" (UniqueName: \"kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-kube-api-access-6x795\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:21.730458 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.730417    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pfsl\" (UniqueName: \"kubernetes.io/projected/353ebfb4-7e11-41de-b992-277443f56960-kube-api-access-2pfsl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-62bhw\" (UID: \"353ebfb4-7e11-41de-b992-277443f56960\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-62bhw"
Oct 30 19:16:21.730530 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.730493    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lksw9\" (UniqueName: \"kubernetes.io/projected/36ab52a2-65e2-4af6-a872-9052e12714d1-kube-api-access-lksw9\") pod \"node-resolver-xjjqh\" (UID: \"36ab52a2-65e2-4af6-a872-9052e12714d1\") " pod="openshift-dns/node-resolver-xjjqh"
Oct 30 19:16:21.730563 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.730513    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tngcd\" (UniqueName: \"kubernetes.io/projected/f1ea27b2-80df-404c-ac39-460f495957b2-kube-api-access-tngcd\") pod \"node-ca-zj8d6\" (UID: \"f1ea27b2-80df-404c-ac39-460f495957b2\") " pod="openshift-image-registry/node-ca-zj8d6"
Oct 30 19:16:21.731174 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.731156    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpb5t\" (UniqueName: \"kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-kube-api-access-hpb5t\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:21.731267 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.731250    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tnx8\" (UniqueName: \"kubernetes.io/projected/c243f1d8-1c43-4eb8-93ec-cb28d72cc8a1-kube-api-access-4tnx8\") pod \"volume-data-source-validator-58c9f5f46d-mg98j\" (UID: \"c243f1d8-1c43-4eb8-93ec-cb28d72cc8a1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-mg98j"
Oct 30 19:16:21.732190 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.732136    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-bound-sa-token\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:21.732565 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.732530    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njpgh\" (UniqueName: \"kubernetes.io/projected/765d8d9d-b89f-419f-b237-3f8034db1bed-kube-api-access-njpgh\") pod \"ingress-canary-9jkcb\" (UID: \"765d8d9d-b89f-419f-b237-3f8034db1bed\") " pod="openshift-ingress-canary/ingress-canary-9jkcb"
Oct 30 19:16:21.732784 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.732718    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s56kf\" (UniqueName: \"kubernetes.io/projected/68201777-2ec5-43b2-b587-81e585865d08-kube-api-access-s56kf\") pod \"service-ca-operator-544d749c8c-vd8tj\" (UID: \"68201777-2ec5-43b2-b587-81e585865d08\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vd8tj"
Oct 30 19:16:21.733043 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.733024    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xrrr\" (UniqueName: \"kubernetes.io/projected/5338b461-6b7c-474b-8458-d1b53775ee0f-kube-api-access-4xrrr\") pod \"periodic-gathering-2lcwk-gjv8f\" (UID: \"5338b461-6b7c-474b-8458-d1b53775ee0f\") " pod="openshift-insights/periodic-gathering-2lcwk-gjv8f"
Oct 30 19:16:21.733241 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.733223    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxkzk\" (UniqueName: \"kubernetes.io/projected/128a8dad-66f8-414b-8c1d-14c7d3b58d25-kube-api-access-jxkzk\") pod \"aws-ebs-csi-driver-node-6k5kr\" (UID: \"128a8dad-66f8-414b-8c1d-14c7d3b58d25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr"
Oct 30 19:16:21.733594 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.733574    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vk66\" (UniqueName: \"kubernetes.io/projected/a37bca55-21cf-4b76-9c05-e2c951647bf8-kube-api-access-2vk66\") pod \"service-ca-5fffc97bdf-b8cs2\" (UID: \"a37bca55-21cf-4b76-9c05-e2c951647bf8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-b8cs2"
Oct 30 19:16:21.733655 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.733593    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6bsd\" (UniqueName: \"kubernetes.io/projected/0269ffa6-ce06-43f3-be2a-58f6aa3e6964-kube-api-access-b6bsd\") pod \"dns-default-gkn6q\" (UID: \"0269ffa6-ce06-43f3-be2a-58f6aa3e6964\") " pod="openshift-dns/dns-default-gkn6q"
Oct 30 19:16:21.733655 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.733603    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8ptq\" (UniqueName: \"kubernetes.io/projected/77fbf490-7af3-4bde-8222-4038e2219fff-kube-api-access-l8ptq\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:16:21.733764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.733676    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rz2d\" (UniqueName: \"kubernetes.io/projected/e5002691-8257-4dab-a88b-3c1b60e8e6f8-kube-api-access-9rz2d\") pod \"insights-operator-7c4986d449-fz9hf\" (UID: \"e5002691-8257-4dab-a88b-3c1b60e8e6f8\") " pod="openshift-insights/insights-operator-7c4986d449-fz9hf"
Oct 30 19:16:21.733764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.733752    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsjcc\" (UniqueName: \"kubernetes.io/projected/c84a5c22-7a46-4de1-aef7-32cb9a5453bb-kube-api-access-zsjcc\") pod \"kas-connection-checker-685895c7c9-zqvrw\" (UID: \"c84a5c22-7a46-4de1-aef7-32cb9a5453bb\") " pod="kube-system/kas-connection-checker-685895c7c9-zqvrw"
Oct 30 19:16:21.734033 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.734018    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbndd\" (UniqueName: \"kubernetes.io/projected/4dbf226e-336f-42c6-a766-cc0ddfe68bef-kube-api-access-lbndd\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:16:21.734089 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.734031    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5c96\" (UniqueName: \"kubernetes.io/projected/dda4158a-327b-4b73-8673-3df1c1c0e400-kube-api-access-m5c96\") pod \"multus-additional-cni-plugins-w2262\" (UID: \"dda4158a-327b-4b73-8673-3df1c1c0e400\") " pod="openshift-multus/multus-additional-cni-plugins-w2262"
Oct 30 19:16:21.822570 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.822527    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-multus-cni-dir\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:16:21.822570 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.822564    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-host-slash\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.822802 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.822587    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6ecb643b-1dce-4523-89df-93b22ad8ce6f-kubelet-config\") pod \"global-pull-secret-syncer-wg2df\" (UID: \"6ecb643b-1dce-4523-89df-93b22ad8ce6f\") " pod="kube-system/global-pull-secret-syncer-wg2df"
Oct 30 19:16:21.822802 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.822609    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-host-run-netns\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:16:21.822802 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.822659    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-host-run-netns\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:16:21.822802 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.822663    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-multus-cni-dir\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:16:21.822802 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.822666    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-host-slash\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.822802 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.822678    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-system-cni-dir\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:16:21.822802 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.822717    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-host-cni-netd\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.822802 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.822680    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6ecb643b-1dce-4523-89df-93b22ad8ce6f-kubelet-config\") pod \"global-pull-secret-syncer-wg2df\" (UID: \"6ecb643b-1dce-4523-89df-93b22ad8ce6f\") " pod="kube-system/global-pull-secret-syncer-wg2df"
Oct 30 19:16:21.822802 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.822785    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-host-cni-netd\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.822802 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.822804    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-etc-modprobe-d\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:16:21.823192 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.822792    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-system-cni-dir\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:16:21.823192 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.822825    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-registry-tls\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:21.823192 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.822871    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-host-run-ovn-kubernetes\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.823192 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.822900    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-etc-modprobe-d\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:16:21.823192 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.822897    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-host-var-lib-cni-multus\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:16:21.823192 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.822933    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-host-var-lib-cni-multus\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:16:21.823192 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.822939    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-host-run-ovn-kubernetes\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.823192 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:21.822940    5043 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:16:21.823192 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.822952    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/06397fd4-3377-4eaa-8879-ebbb5943fbda-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2sdkw\" (UID: \"06397fd4-3377-4eaa-8879-ebbb5943fbda\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2sdkw"
Oct 30 19:16:21.823192 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:21.822963    5043 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-748f8689f4-6lwjn: secret "image-registry-tls" not found
Oct 30 19:16:21.823192 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.822972    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-host-run-multus-certs\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:16:21.823192 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.822990    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.823192 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823006    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f1ea27b2-80df-404c-ac39-460f495957b2-host\") pod \"node-ca-zj8d6\" (UID: \"f1ea27b2-80df-404c-ac39-460f495957b2\") " pod="openshift-image-registry/node-ca-zj8d6"
Oct 30 19:16:21.823192 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:21.823033    5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-registry-tls podName:9f302411-01be-438c-a30c-70c944afd630 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:22.323017237 +0000 UTC m=+1.834077202 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-registry-tls") pod "image-registry-748f8689f4-6lwjn" (UID: "9f302411-01be-438c-a30c-70c944afd630") : secret "image-registry-tls" not found
Oct 30 19:16:21.823192 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823040    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-host-run-multus-certs\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:16:21.823192 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823046    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f1ea27b2-80df-404c-ac39-460f495957b2-host\") pod \"node-ca-zj8d6\" (UID: \"f1ea27b2-80df-404c-ac39-460f495957b2\") " pod="openshift-image-registry/node-ca-zj8d6"
Oct 30 19:16:21.823192 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823047    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.823192 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823046    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-host-run-k8s-cni-cncf-io\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:16:21.823192 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823068    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-host-run-k8s-cni-cncf-io\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:16:21.823192 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823099    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-systemd-units\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.823192 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:21.823110    5043 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:16:21.823192 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823123    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-systemd-units\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.823192 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823139    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dda4158a-327b-4b73-8673-3df1c1c0e400-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w2262\" (UID: \"dda4158a-327b-4b73-8673-3df1c1c0e400\") " pod="openshift-multus/multus-additional-cni-plugins-w2262"
Oct 30 19:16:21.823192 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823159    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-etc-openvswitch\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.823192 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:21.823185    5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06397fd4-3377-4eaa-8879-ebbb5943fbda-cluster-monitoring-operator-tls podName:06397fd4-3377-4eaa-8879-ebbb5943fbda nodeName:}" failed. No retries permitted until 2026-10-30 19:16:22.323166593 +0000 UTC m=+1.834226557 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/06397fd4-3377-4eaa-8879-ebbb5943fbda-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2sdkw" (UID: "06397fd4-3377-4eaa-8879-ebbb5943fbda") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:16:21.823192 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823187    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-etc-openvswitch\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.823192 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823205    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-host-cni-bin\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823219    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dda4158a-327b-4b73-8673-3df1c1c0e400-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w2262\" (UID: \"dda4158a-327b-4b73-8673-3df1c1c0e400\") " pod="openshift-multus/multus-additional-cni-plugins-w2262"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823240    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/128a8dad-66f8-414b-8c1d-14c7d3b58d25-registration-dir\") pod \"aws-ebs-csi-driver-node-6k5kr\" (UID: \"128a8dad-66f8-414b-8c1d-14c7d3b58d25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823268    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-hostroot\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823285    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/128a8dad-66f8-414b-8c1d-14c7d3b58d25-socket-dir\") pod \"aws-ebs-csi-driver-node-6k5kr\" (UID: \"128a8dad-66f8-414b-8c1d-14c7d3b58d25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823241    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-host-cni-bin\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823297    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/128a8dad-66f8-414b-8c1d-14c7d3b58d25-registration-dir\") pod \"aws-ebs-csi-driver-node-6k5kr\" (UID: \"128a8dad-66f8-414b-8c1d-14c7d3b58d25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823317    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-lib-modules\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823334    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/128a8dad-66f8-414b-8c1d-14c7d3b58d25-socket-dir\") pod \"aws-ebs-csi-driver-node-6k5kr\" (UID: \"128a8dad-66f8-414b-8c1d-14c7d3b58d25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823343    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-hostroot\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823377    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dda4158a-327b-4b73-8673-3df1c1c0e400-cnibin\") pod \"multus-additional-cni-plugins-w2262\" (UID: \"dda4158a-327b-4b73-8673-3df1c1c0e400\") " pod="openshift-multus/multus-additional-cni-plugins-w2262"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823403    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0269ffa6-ce06-43f3-be2a-58f6aa3e6964-metrics-tls\") pod \"dns-default-gkn6q\" (UID: \"0269ffa6-ce06-43f3-be2a-58f6aa3e6964\") " pod="openshift-dns/dns-default-gkn6q"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823417    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-run\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823435    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aeef13dc-e06e-49c5-b893-64c7f73a7e44-crio-socket\") pod \"insights-runtime-extractor-smgf6\" (UID: \"aeef13dc-e06e-49c5-b893-64c7f73a7e44\") " pod="openshift-insights/insights-runtime-extractor-smgf6"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823444    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-lib-modules\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823450    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-var-lib-kubelet\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823481    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dda4158a-327b-4b73-8673-3df1c1c0e400-cnibin\") pod \"multus-additional-cni-plugins-w2262\" (UID: \"dda4158a-327b-4b73-8673-3df1c1c0e400\") " pod="openshift-multus/multus-additional-cni-plugins-w2262"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823497    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-run\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823506    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-var-lib-kubelet\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:21.823526    5043 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823559    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aeef13dc-e06e-49c5-b893-64c7f73a7e44-crio-socket\") pod \"insights-runtime-extractor-smgf6\" (UID: \"aeef13dc-e06e-49c5-b893-64c7f73a7e44\") " pod="openshift-insights/insights-runtime-extractor-smgf6"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:21.823611    5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0269ffa6-ce06-43f3-be2a-58f6aa3e6964-metrics-tls podName:0269ffa6-ce06-43f3-be2a-58f6aa3e6964 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:22.323596708 +0000 UTC m=+1.834656672 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0269ffa6-ce06-43f3-be2a-58f6aa3e6964-metrics-tls") pod "dns-default-gkn6q" (UID: "0269ffa6-ce06-43f3-be2a-58f6aa3e6964") : secret "dns-default-metrics-tls" not found
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823527    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aeef13dc-e06e-49c5-b893-64c7f73a7e44-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-smgf6\" (UID: \"aeef13dc-e06e-49c5-b893-64c7f73a7e44\") " pod="openshift-insights/insights-runtime-extractor-smgf6"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823635    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aeef13dc-e06e-49c5-b893-64c7f73a7e44-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-smgf6\" (UID: \"aeef13dc-e06e-49c5-b893-64c7f73a7e44\") " pod="openshift-insights/insights-runtime-extractor-smgf6"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823651    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-cnibin\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823666    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-etc-kubernetes\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823682    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/128a8dad-66f8-414b-8c1d-14c7d3b58d25-device-dir\") pod \"aws-ebs-csi-driver-node-6k5kr\" (UID: \"128a8dad-66f8-414b-8c1d-14c7d3b58d25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823696    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dda4158a-327b-4b73-8673-3df1c1c0e400-system-cni-dir\") pod \"multus-additional-cni-plugins-w2262\" (UID: \"dda4158a-327b-4b73-8673-3df1c1c0e400\") " pod="openshift-multus/multus-additional-cni-plugins-w2262"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:21.823700    5043 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823710    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cbc4f7c7-bebd-4200-833e-1afbf3c2c81a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-p5nzf\" (UID: \"cbc4f7c7-bebd-4200-833e-1afbf3c2c81a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p5nzf"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823724    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-host-var-lib-cni-bin\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823762    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-etc-kubernetes\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823769    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-node-log\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823780    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dda4158a-327b-4b73-8673-3df1c1c0e400-system-cni-dir\") pod \"multus-additional-cni-plugins-w2262\" (UID: \"dda4158a-327b-4b73-8673-3df1c1c0e400\") " pod="openshift-multus/multus-additional-cni-plugins-w2262"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823796    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-cnibin\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:21.823811    5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeef13dc-e06e-49c5-b893-64c7f73a7e44-insights-runtime-extractor-tls podName:aeef13dc-e06e-49c5-b893-64c7f73a7e44 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:22.323798236 +0000 UTC m=+1.834858181 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aeef13dc-e06e-49c5-b893-64c7f73a7e44-insights-runtime-extractor-tls") pod "insights-runtime-extractor-smgf6" (UID: "aeef13dc-e06e-49c5-b893-64c7f73a7e44") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823808    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-host-var-lib-cni-bin\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823798    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/128a8dad-66f8-414b-8c1d-14c7d3b58d25-device-dir\") pod \"aws-ebs-csi-driver-node-6k5kr\" (UID: \"128a8dad-66f8-414b-8c1d-14c7d3b58d25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823828    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-host-var-lib-kubelet\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823839    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-host-var-lib-kubelet\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823861    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a85cca66-a438-4117-9f50-977c7f528b7c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lpg7w\" (UID: \"a85cca66-a438-4117-9f50-977c7f528b7c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lpg7w"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:21.823868    5043 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823815    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-node-log\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823886    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-host\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823911    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/36ab52a2-65e2-4af6-a872-9052e12714d1-hosts-file\") pod \"node-resolver-xjjqh\" (UID: \"36ab52a2-65e2-4af6-a872-9052e12714d1\") " pod="openshift-dns/node-resolver-xjjqh"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:21.823926    5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbc4f7c7-bebd-4200-833e-1afbf3c2c81a-samples-operator-tls podName:cbc4f7c7-bebd-4200-833e-1afbf3c2c81a nodeName:}" failed. No retries permitted until 2026-10-30 19:16:22.323915513 +0000 UTC m=+1.834975469 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cbc4f7c7-bebd-4200-833e-1afbf3c2c81a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-p5nzf" (UID: "cbc4f7c7-bebd-4200-833e-1afbf3c2c81a") : secret "samples-operator-tls" not found
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:21.823942    5043 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823949    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/128a8dad-66f8-414b-8c1d-14c7d3b58d25-etc-selinux\") pod \"aws-ebs-csi-driver-node-6k5kr\" (UID: \"128a8dad-66f8-414b-8c1d-14c7d3b58d25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823953    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/36ab52a2-65e2-4af6-a872-9052e12714d1-hosts-file\") pod \"node-resolver-xjjqh\" (UID: \"36ab52a2-65e2-4af6-a872-9052e12714d1\") " pod="openshift-dns/node-resolver-xjjqh"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823973    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-host\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:21.823985    5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a85cca66-a438-4117-9f50-977c7f528b7c-networking-console-plugin-cert podName:a85cca66-a438-4117-9f50-977c7f528b7c nodeName:}" failed. No retries permitted until 2026-10-30 19:16:22.323971727 +0000 UTC m=+1.835031697 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a85cca66-a438-4117-9f50-977c7f528b7c-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-lpg7w" (UID: "a85cca66-a438-4117-9f50-977c7f528b7c") : secret "networking-console-plugin-cert" not found
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.823977    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-etc-kubernetes\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.824014    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-etc-kubernetes\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.824012    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/128a8dad-66f8-414b-8c1d-14c7d3b58d25-etc-selinux\") pod \"aws-ebs-csi-driver-node-6k5kr\" (UID: \"128a8dad-66f8-414b-8c1d-14c7d3b58d25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.824019    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-etc-sysctl-conf\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.824035    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aeef13dc-e06e-49c5-b893-64c7f73a7e44-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-smgf6\" (UID: \"aeef13dc-e06e-49c5-b893-64c7f73a7e44\") " pod="openshift-insights/insights-runtime-extractor-smgf6"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.824046    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-etc-systemd\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.824070    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-host-kubelet\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.824080    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-etc-systemd\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.824087    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/128a8dad-66f8-414b-8c1d-14c7d3b58d25-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6k5kr\" (UID: \"128a8dad-66f8-414b-8c1d-14c7d3b58d25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.824106    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-host-kubelet\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.824107    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-etc-sysctl-conf\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.824114    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/128a8dad-66f8-414b-8c1d-14c7d3b58d25-sys-fs\") pod \"aws-ebs-csi-driver-node-6k5kr\" (UID: \"128a8dad-66f8-414b-8c1d-14c7d3b58d25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.824133    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/128a8dad-66f8-414b-8c1d-14c7d3b58d25-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6k5kr\" (UID: \"128a8dad-66f8-414b-8c1d-14c7d3b58d25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.824137    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-etc-sysctl-d\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.824145    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/128a8dad-66f8-414b-8c1d-14c7d3b58d25-sys-fs\") pod \"aws-ebs-csi-driver-node-6k5kr\" (UID: \"128a8dad-66f8-414b-8c1d-14c7d3b58d25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6k5kr"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.824162    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-var-lib-openvswitch\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.824179    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-run-openvswitch\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.824193    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-run-ovn\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.824197    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-var-lib-openvswitch\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.824183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.824212    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-multus-socket-dir-parent\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:16:21.826280 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.824231    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-run-openvswitch\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.826280 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.824237    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-multus-conf-dir\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:16:21.826280 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.824242    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-etc-sysctl-d\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:16:21.826280 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.824247    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-run-ovn\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.826280 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.824261    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/247ff425-bed3-4297-b5c0-cc32b21bc179-metrics-certs\") pod \"router-default-75866dc98c-4mnhf\" (UID: \"247ff425-bed3-4297-b5c0-cc32b21bc179\") " pod="openshift-ingress/router-default-75866dc98c-4mnhf"
Oct 30 19:16:21.826280 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.824275    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-multus-conf-dir\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:16:21.826280 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.824282    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/247ff425-bed3-4297-b5c0-cc32b21bc179-service-ca-bundle\") pod \"router-default-75866dc98c-4mnhf\" (UID: \"247ff425-bed3-4297-b5c0-cc32b21bc179\") " pod="openshift-ingress/router-default-75866dc98c-4mnhf"
Oct 30 19:16:21.826280 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.824265    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4dbf226e-336f-42c6-a766-cc0ddfe68bef-multus-socket-dir-parent\") pod \"multus-jxqzn\" (UID: \"4dbf226e-336f-42c6-a766-cc0ddfe68bef\") " pod="openshift-multus/multus-jxqzn"
Oct 30 19:16:21.826280 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.824299    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-registry-tls\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:21.826280 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.824322    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-run-systemd\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.826280 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.824343    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-sys\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:16:21.826280 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.824368    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8e297a2-a78a-4665-86d0-6d842bbcaa51-metrics-certs\") pod \"network-metrics-daemon-n9cws\" (UID: \"e8e297a2-a78a-4665-86d0-6d842bbcaa51\") " pod="openshift-multus/network-metrics-daemon-n9cws"
Oct 30 19:16:21.826280 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.824385    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-run-systemd\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.826280 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.824396    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-etc-sysconfig\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:16:21.826280 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:21.824372    5043 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:16:21.826280 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.824412    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-sys\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:16:21.826280 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:21.824419    5043 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78dc458d47-tqn8s: secret "image-registry-tls" not found
Oct 30 19:16:21.826280 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.824435    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/77fbf490-7af3-4bde-8222-4038e2219fff-etc-sysconfig\") pod \"tuned-tkdbv\" (UID: \"77fbf490-7af3-4bde-8222-4038e2219fff\") " pod="openshift-cluster-node-tuning-operator/tuned-tkdbv"
Oct 30 19:16:21.826280 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:21.824404    5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/247ff425-bed3-4297-b5c0-cc32b21bc179-service-ca-bundle podName:247ff425-bed3-4297-b5c0-cc32b21bc179 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:22.32439367 +0000 UTC m=+1.835453638 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/247ff425-bed3-4297-b5c0-cc32b21bc179-service-ca-bundle") pod "router-default-75866dc98c-4mnhf" (UID: "247ff425-bed3-4297-b5c0-cc32b21bc179") : configmap references non-existent config key: service-ca.crt
Oct 30 19:16:21.826280 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.824494    5043 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 19:16:21.826280 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:21.824532    5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-registry-tls podName:a64211d0-1c48-41d2-b254-4ae84d5097d7 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:22.324524875 +0000 UTC m=+1.835584820 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-registry-tls") pod "image-registry-78dc458d47-tqn8s" (UID: "a64211d0-1c48-41d2-b254-4ae84d5097d7") : secret "image-registry-tls" not found
Oct 30 19:16:21.826280 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.824544    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-host-run-netns\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.826280 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.824561    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-log-socket\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.826280 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.824586    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6331f51e-cf5e-4ba2-9eda-35fd62b14091-host-slash\") pod \"iptables-alerter-26xq4\" (UID: \"6331f51e-cf5e-4ba2-9eda-35fd62b14091\") " pod="openshift-network-operator/iptables-alerter-26xq4"
Oct 30 19:16:21.826280 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.824609    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-host-run-netns\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.826280 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.824611    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6ecb643b-1dce-4523-89df-93b22ad8ce6f-dbus\") pod \"global-pull-secret-syncer-wg2df\" (UID: \"6ecb643b-1dce-4523-89df-93b22ad8ce6f\") " pod="kube-system/global-pull-secret-syncer-wg2df"
Oct 30 19:16:21.826280 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.824627    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/dc581227-9542-4571-aced-63d191692a41-log-socket\") pod \"ovnkube-node-s8t94\" (UID: \"dc581227-9542-4571-aced-63d191692a41\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:21.826280 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.824657    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6ecb643b-1dce-4523-89df-93b22ad8ce6f-dbus\") pod \"global-pull-secret-syncer-wg2df\" (UID: \"6ecb643b-1dce-4523-89df-93b22ad8ce6f\") " pod="kube-system/global-pull-secret-syncer-wg2df"
Oct 30 19:16:21.826280 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.824690    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6331f51e-cf5e-4ba2-9eda-35fd62b14091-host-slash\") pod \"iptables-alerter-26xq4\" (UID: \"6331f51e-cf5e-4ba2-9eda-35fd62b14091\") " pod="openshift-network-operator/iptables-alerter-26xq4"
Oct 30 19:16:21.827125 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.826795    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8e297a2-a78a-4665-86d0-6d842bbcaa51-metrics-certs\") pod \"network-metrics-daemon-n9cws\" (UID: \"e8e297a2-a78a-4665-86d0-6d842bbcaa51\") " pod="openshift-multus/network-metrics-daemon-n9cws"
Oct 30 19:16:21.827125 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.826922    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/247ff425-bed3-4297-b5c0-cc32b21bc179-metrics-certs\") pod \"router-default-75866dc98c-4mnhf\" (UID: \"247ff425-bed3-4297-b5c0-cc32b21bc179\") " pod="openshift-ingress/router-default-75866dc98c-4mnhf"
Oct 30 19:16:21.904688 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.904612    5043 scope.go:122] "RemoveContainer" containerID="b3f38d74f7964cb2e0db345a5bf20e6f2b18ed1ffffa37aafe2a406f3f91103c"
Oct 30 19:16:21.918484 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:21.918461    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9cws"
Oct 30 19:16:22.184440 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:22.184357    5043 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-cmw6w"
Oct 30 19:16:22.211484 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:22.211455    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n9cws"]
Oct 30 19:16:22.214809 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:22.214774    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8e297a2_a78a_4665_86d0_6d842bbcaa51.slice/crio-74b5245367a3df2c12eb9edc4561ad1f1aa9f64be87ee94bb62ec34ab864e19a WatchSource:0}: Error finding container 74b5245367a3df2c12eb9edc4561ad1f1aa9f64be87ee94bb62ec34ab864e19a: Status 404 returned error can't find the container with id 74b5245367a3df2c12eb9edc4561ad1f1aa9f64be87ee94bb62ec34ab864e19a
Oct 30 19:16:22.216919 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:22.216904    5043 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:16:22.329164 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:22.329135    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aeef13dc-e06e-49c5-b893-64c7f73a7e44-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-smgf6\" (UID: \"aeef13dc-e06e-49c5-b893-64c7f73a7e44\") " pod="openshift-insights/insights-runtime-extractor-smgf6"
Oct 30 19:16:22.329164 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:22.329170    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cbc4f7c7-bebd-4200-833e-1afbf3c2c81a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-p5nzf\" (UID: \"cbc4f7c7-bebd-4200-833e-1afbf3c2c81a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p5nzf"
Oct 30 19:16:22.329404 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:22.329194    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a85cca66-a438-4117-9f50-977c7f528b7c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lpg7w\" (UID: \"a85cca66-a438-4117-9f50-977c7f528b7c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lpg7w"
Oct 30 19:16:22.329404 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:22.329224    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/247ff425-bed3-4297-b5c0-cc32b21bc179-service-ca-bundle\") pod \"router-default-75866dc98c-4mnhf\" (UID: \"247ff425-bed3-4297-b5c0-cc32b21bc179\") " pod="openshift-ingress/router-default-75866dc98c-4mnhf"
Oct 30 19:16:22.329404 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:22.329240    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-registry-tls\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:22.329404 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:22.329279    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-registry-tls\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:22.329404 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:22.329297    5043 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:16:22.329404 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:22.329298    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/06397fd4-3377-4eaa-8879-ebbb5943fbda-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2sdkw\" (UID: \"06397fd4-3377-4eaa-8879-ebbb5943fbda\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2sdkw"
Oct 30 19:16:22.329404 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:22.329316    5043 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:16:22.329404 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:22.329367    5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeef13dc-e06e-49c5-b893-64c7f73a7e44-insights-runtime-extractor-tls podName:aeef13dc-e06e-49c5-b893-64c7f73a7e44 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:23.32935211 +0000 UTC m=+2.840412070 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aeef13dc-e06e-49c5-b893-64c7f73a7e44-insights-runtime-extractor-tls") pod "insights-runtime-extractor-smgf6" (UID: "aeef13dc-e06e-49c5-b893-64c7f73a7e44") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:16:22.329404 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:22.329382    5043 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:16:22.329404 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:22.329403    5043 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-748f8689f4-6lwjn: secret "image-registry-tls" not found
Oct 30 19:16:22.329404 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:22.329408    5043 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:16:22.329956 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:22.329316    5043 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 19:16:22.329956 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:22.329382    5043 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:16:22.329956 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:22.329470    5043 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78dc458d47-tqn8s: secret "image-registry-tls" not found
Oct 30 19:16:22.329956 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:22.329386    5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/247ff425-bed3-4297-b5c0-cc32b21bc179-service-ca-bundle podName:247ff425-bed3-4297-b5c0-cc32b21bc179 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:23.329380556 +0000 UTC m=+2.840440498 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/247ff425-bed3-4297-b5c0-cc32b21bc179-service-ca-bundle") pod "router-default-75866dc98c-4mnhf" (UID: "247ff425-bed3-4297-b5c0-cc32b21bc179") : configmap references non-existent config key: service-ca.crt
Oct 30 19:16:22.329956 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:22.329508    5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a85cca66-a438-4117-9f50-977c7f528b7c-networking-console-plugin-cert podName:a85cca66-a438-4117-9f50-977c7f528b7c nodeName:}" failed. No retries permitted until 2026-10-30 19:16:23.329494628 +0000 UTC m=+2.840554571 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a85cca66-a438-4117-9f50-977c7f528b7c-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-lpg7w" (UID: "a85cca66-a438-4117-9f50-977c7f528b7c") : secret "networking-console-plugin-cert" not found
Oct 30 19:16:22.329956 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:22.329520    5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06397fd4-3377-4eaa-8879-ebbb5943fbda-cluster-monitoring-operator-tls podName:06397fd4-3377-4eaa-8879-ebbb5943fbda nodeName:}" failed. No retries permitted until 2026-10-30 19:16:23.329515038 +0000 UTC m=+2.840574981 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/06397fd4-3377-4eaa-8879-ebbb5943fbda-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2sdkw" (UID: "06397fd4-3377-4eaa-8879-ebbb5943fbda") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:16:22.329956 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:22.329540    5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-registry-tls podName:9f302411-01be-438c-a30c-70c944afd630 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:23.329533961 +0000 UTC m=+2.840593916 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-registry-tls") pod "image-registry-748f8689f4-6lwjn" (UID: "9f302411-01be-438c-a30c-70c944afd630") : secret "image-registry-tls" not found
Oct 30 19:16:22.329956 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:22.329554    5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbc4f7c7-bebd-4200-833e-1afbf3c2c81a-samples-operator-tls podName:cbc4f7c7-bebd-4200-833e-1afbf3c2c81a nodeName:}" failed. No retries permitted until 2026-10-30 19:16:23.329548785 +0000 UTC m=+2.840608728 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cbc4f7c7-bebd-4200-833e-1afbf3c2c81a-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-p5nzf" (UID: "cbc4f7c7-bebd-4200-833e-1afbf3c2c81a") : secret "samples-operator-tls" not found
Oct 30 19:16:22.329956 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:22.329567    5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-registry-tls podName:a64211d0-1c48-41d2-b254-4ae84d5097d7 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:23.329562321 +0000 UTC m=+2.840622264 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-registry-tls") pod "image-registry-78dc458d47-tqn8s" (UID: "a64211d0-1c48-41d2-b254-4ae84d5097d7") : secret "image-registry-tls" not found
Oct 30 19:16:22.329956 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:22.329583    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0269ffa6-ce06-43f3-be2a-58f6aa3e6964-metrics-tls\") pod \"dns-default-gkn6q\" (UID: \"0269ffa6-ce06-43f3-be2a-58f6aa3e6964\") " pod="openshift-dns/dns-default-gkn6q"
Oct 30 19:16:22.329956 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:22.329652    5043 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:16:22.329956 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:22.329678    5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0269ffa6-ce06-43f3-be2a-58f6aa3e6964-metrics-tls podName:0269ffa6-ce06-43f3-be2a-58f6aa3e6964 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:23.329672011 +0000 UTC m=+2.840731973 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0269ffa6-ce06-43f3-be2a-58f6aa3e6964-metrics-tls") pod "dns-default-gkn6q" (UID: "0269ffa6-ce06-43f3-be2a-58f6aa3e6964") : secret "dns-default-metrics-tls" not found
Oct 30 19:16:22.702171 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:22.702134    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n9cws" event={"ID":"e8e297a2-a78a-4665-86d0-6d842bbcaa51","Type":"ContainerStarted","Data":"74b5245367a3df2c12eb9edc4561ad1f1aa9f64be87ee94bb62ec34ab864e19a"}
Oct 30 19:16:22.704150 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:22.704123    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cmw6w_76e21916-0744-4499-a0cb-64a7a44a8285/console-operator/2.log"
Oct 30 19:16:22.704632 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:22.704614    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cmw6w_76e21916-0744-4499-a0cb-64a7a44a8285/console-operator/1.log"
Oct 30 19:16:22.704855 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:22.704648    5043 generic.go:356] "Generic (PLEG): container finished" podID="76e21916-0744-4499-a0cb-64a7a44a8285" containerID="981122b0f414c61dfba8d729d8a7e129d805b2486a8ad704c2cf06fb773ab2d4" exitCode=255
Oct 30 19:16:22.704855 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:22.704707    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-cmw6w" event={"ID":"76e21916-0744-4499-a0cb-64a7a44a8285","Type":"ContainerDied","Data":"981122b0f414c61dfba8d729d8a7e129d805b2486a8ad704c2cf06fb773ab2d4"}
Oct 30 19:16:22.704855 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:22.704786    5043 scope.go:122] "RemoveContainer" containerID="b3f38d74f7964cb2e0db345a5bf20e6f2b18ed1ffffa37aafe2a406f3f91103c"
Oct 30 19:16:22.705076 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:22.705051    5043 scope.go:122] "RemoveContainer" containerID="981122b0f414c61dfba8d729d8a7e129d805b2486a8ad704c2cf06fb773ab2d4"
Oct 30 19:16:22.705319 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:22.705296    5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-cmw6w_openshift-console-operator(76e21916-0744-4499-a0cb-64a7a44a8285)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-cmw6w" podUID="76e21916-0744-4499-a0cb-64a7a44a8285"
Oct 30 19:16:23.098981 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:23.098950    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:23.115429 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:23.115393    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:23.339297 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:23.339264    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-registry-tls\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:23.339415 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:23.339321    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/06397fd4-3377-4eaa-8879-ebbb5943fbda-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2sdkw\" (UID: \"06397fd4-3377-4eaa-8879-ebbb5943fbda\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2sdkw"
Oct 30 19:16:23.339415 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:23.339371    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0269ffa6-ce06-43f3-be2a-58f6aa3e6964-metrics-tls\") pod \"dns-default-gkn6q\" (UID: \"0269ffa6-ce06-43f3-be2a-58f6aa3e6964\") " pod="openshift-dns/dns-default-gkn6q"
Oct 30 19:16:23.339415 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:23.339393    5043 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:16:23.339415 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:23.339401    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aeef13dc-e06e-49c5-b893-64c7f73a7e44-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-smgf6\" (UID: \"aeef13dc-e06e-49c5-b893-64c7f73a7e44\") " pod="openshift-insights/insights-runtime-extractor-smgf6"
Oct 30 19:16:23.339415 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:23.339413    5043 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-748f8689f4-6lwjn: secret "image-registry-tls" not found
Oct 30 19:16:23.339671 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:23.339431    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cbc4f7c7-bebd-4200-833e-1afbf3c2c81a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-p5nzf\" (UID: \"cbc4f7c7-bebd-4200-833e-1afbf3c2c81a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p5nzf"
Oct 30 19:16:23.339671 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:23.339459    5043 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:16:23.339671 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:23.339471    5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-registry-tls podName:9f302411-01be-438c-a30c-70c944afd630 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:25.339452185 +0000 UTC m=+4.850512135 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-registry-tls") pod "image-registry-748f8689f4-6lwjn" (UID: "9f302411-01be-438c-a30c-70c944afd630") : secret "image-registry-tls" not found
Oct 30 19:16:23.339671 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:23.339494    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a85cca66-a438-4117-9f50-977c7f528b7c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lpg7w\" (UID: \"a85cca66-a438-4117-9f50-977c7f528b7c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lpg7w"
Oct 30 19:16:23.339671 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:23.339506    5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06397fd4-3377-4eaa-8879-ebbb5943fbda-cluster-monitoring-operator-tls podName:06397fd4-3377-4eaa-8879-ebbb5943fbda nodeName:}" failed. No retries permitted until 2026-10-30 19:16:25.339494353 +0000 UTC m=+4.850554300 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/06397fd4-3377-4eaa-8879-ebbb5943fbda-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2sdkw" (UID: "06397fd4-3377-4eaa-8879-ebbb5943fbda") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:16:23.339671 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:23.339556    5043 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:16:23.339671 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:23.339589    5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a85cca66-a438-4117-9f50-977c7f528b7c-networking-console-plugin-cert podName:a85cca66-a438-4117-9f50-977c7f528b7c nodeName:}" failed. No retries permitted until 2026-10-30 19:16:25.339578913 +0000 UTC m=+4.850638857 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a85cca66-a438-4117-9f50-977c7f528b7c-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-lpg7w" (UID: "a85cca66-a438-4117-9f50-977c7f528b7c") : secret "networking-console-plugin-cert" not found
Oct 30 19:16:23.339671 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:23.339555    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/247ff425-bed3-4297-b5c0-cc32b21bc179-service-ca-bundle\") pod \"router-default-75866dc98c-4mnhf\" (UID: \"247ff425-bed3-4297-b5c0-cc32b21bc179\") " pod="openshift-ingress/router-default-75866dc98c-4mnhf"
Oct 30 19:16:23.339671 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:23.339609    5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/247ff425-bed3-4297-b5c0-cc32b21bc179-service-ca-bundle podName:247ff425-bed3-4297-b5c0-cc32b21bc179 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:25.339601922 +0000 UTC m=+4.850661864 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/247ff425-bed3-4297-b5c0-cc32b21bc179-service-ca-bundle") pod "router-default-75866dc98c-4mnhf" (UID: "247ff425-bed3-4297-b5c0-cc32b21bc179") : configmap references non-existent config key: service-ca.crt
Oct 30 19:16:23.339671 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:23.339623    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-registry-tls\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:23.340135 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:23.339720    5043 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:16:23.340135 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:23.339728    5043 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78dc458d47-tqn8s: secret "image-registry-tls" not found
Oct 30 19:16:23.340135 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:23.339770    5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-registry-tls podName:a64211d0-1c48-41d2-b254-4ae84d5097d7 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:25.339763007 +0000 UTC m=+4.850822950 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-registry-tls") pod "image-registry-78dc458d47-tqn8s" (UID: "a64211d0-1c48-41d2-b254-4ae84d5097d7") : secret "image-registry-tls" not found
Oct 30 19:16:23.340135 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:23.339814    5043 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:16:23.340135 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:23.339841    5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeef13dc-e06e-49c5-b893-64c7f73a7e44-insights-runtime-extractor-tls podName:aeef13dc-e06e-49c5-b893-64c7f73a7e44 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:25.339833644 +0000 UTC m=+4.850893588 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aeef13dc-e06e-49c5-b893-64c7f73a7e44-insights-runtime-extractor-tls") pod "insights-runtime-extractor-smgf6" (UID: "aeef13dc-e06e-49c5-b893-64c7f73a7e44") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:16:23.342207 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:23.342185    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0269ffa6-ce06-43f3-be2a-58f6aa3e6964-metrics-tls\") pod \"dns-default-gkn6q\" (UID: \"0269ffa6-ce06-43f3-be2a-58f6aa3e6964\") " pod="openshift-dns/dns-default-gkn6q"
Oct 30 19:16:23.342375 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:23.342353    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cbc4f7c7-bebd-4200-833e-1afbf3c2c81a-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-p5nzf\" (UID: \"cbc4f7c7-bebd-4200-833e-1afbf3c2c81a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p5nzf"
Oct 30 19:16:23.404279 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:23.404196    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p5nzf"
Oct 30 19:16:23.418714 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:23.418688    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gkn6q"
Oct 30 19:16:23.677770 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:23.677681    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p5nzf"]
Oct 30 19:16:23.701480 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:23.701456    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gkn6q"]
Oct 30 19:16:23.705842 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:23.705799    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0269ffa6_ce06_43f3_be2a_58f6aa3e6964.slice/crio-eb4c1b49bc1d88de12b069a575f4990daa3464aa5226a6c48b5a3c4e755b735a WatchSource:0}: Error finding container eb4c1b49bc1d88de12b069a575f4990daa3464aa5226a6c48b5a3c4e755b735a: Status 404 returned error can't find the container with id eb4c1b49bc1d88de12b069a575f4990daa3464aa5226a6c48b5a3c4e755b735a
Oct 30 19:16:23.710552 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:23.710526    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n9cws" event={"ID":"e8e297a2-a78a-4665-86d0-6d842bbcaa51","Type":"ContainerStarted","Data":"f88ee1e263aa67e2045435d9634df6ff6ec187d434f89acfc32f631f4480ae91"}
Oct 30 19:16:23.710652 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:23.710596    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n9cws" event={"ID":"e8e297a2-a78a-4665-86d0-6d842bbcaa51","Type":"ContainerStarted","Data":"ac6132452fcd08e3ec85c0bc0b427379cbb9bc685b53c13304dbc305d30b672b"}
Oct 30 19:16:23.711922 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:23.711905    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cmw6w_76e21916-0744-4499-a0cb-64a7a44a8285/console-operator/2.log"
Oct 30 19:16:23.712181 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:23.712164    5043 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:16:23.712227 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:23.712177    5043 scope.go:122] "RemoveContainer" containerID="981122b0f414c61dfba8d729d8a7e129d805b2486a8ad704c2cf06fb773ab2d4"
Oct 30 19:16:23.712391 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:23.712371    5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-cmw6w_openshift-console-operator(76e21916-0744-4499-a0cb-64a7a44a8285)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-cmw6w" podUID="76e21916-0744-4499-a0cb-64a7a44a8285"
Oct 30 19:16:24.718116 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:24.718077    5043 generic.go:356] "Generic (PLEG): container finished" podID="e5002691-8257-4dab-a88b-3c1b60e8e6f8" containerID="de80aadc9929788187e54b1f259f5300d637f50bdc7d72b6d616054e2fdd1196" exitCode=0
Oct 30 19:16:24.718525 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:24.718198    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-fz9hf" event={"ID":"e5002691-8257-4dab-a88b-3c1b60e8e6f8","Type":"ContainerDied","Data":"de80aadc9929788187e54b1f259f5300d637f50bdc7d72b6d616054e2fdd1196"}
Oct 30 19:16:24.718612 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:24.718567    5043 scope.go:122] "RemoveContainer" containerID="de80aadc9929788187e54b1f259f5300d637f50bdc7d72b6d616054e2fdd1196"
Oct 30 19:16:24.722182 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:24.722155    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gkn6q" event={"ID":"0269ffa6-ce06-43f3-be2a-58f6aa3e6964","Type":"ContainerStarted","Data":"eb4c1b49bc1d88de12b069a575f4990daa3464aa5226a6c48b5a3c4e755b735a"}
Oct 30 19:16:24.724765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:24.724721    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p5nzf" event={"ID":"cbc4f7c7-bebd-4200-833e-1afbf3c2c81a","Type":"ContainerStarted","Data":"3c1dfad9e5b940a95f2babb4aaaa34f44d499d1c265c26e4c7c16cbce7122529"}
Oct 30 19:16:25.359851 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:25.359812    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a85cca66-a438-4117-9f50-977c7f528b7c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lpg7w\" (UID: \"a85cca66-a438-4117-9f50-977c7f528b7c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lpg7w"
Oct 30 19:16:25.360045 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:25.359883    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/247ff425-bed3-4297-b5c0-cc32b21bc179-service-ca-bundle\") pod \"router-default-75866dc98c-4mnhf\" (UID: \"247ff425-bed3-4297-b5c0-cc32b21bc179\") " pod="openshift-ingress/router-default-75866dc98c-4mnhf"
Oct 30 19:16:25.360045 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:25.359922    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-registry-tls\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:25.360045 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:25.359989    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-registry-tls\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:25.360045 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:25.360023    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/06397fd4-3377-4eaa-8879-ebbb5943fbda-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2sdkw\" (UID: \"06397fd4-3377-4eaa-8879-ebbb5943fbda\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2sdkw"
Oct 30 19:16:25.360250 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:25.360079    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aeef13dc-e06e-49c5-b893-64c7f73a7e44-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-smgf6\" (UID: \"aeef13dc-e06e-49c5-b893-64c7f73a7e44\") " pod="openshift-insights/insights-runtime-extractor-smgf6"
Oct 30 19:16:25.360250 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:25.360220    5043 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:16:25.360341 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:25.360280    5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeef13dc-e06e-49c5-b893-64c7f73a7e44-insights-runtime-extractor-tls podName:aeef13dc-e06e-49c5-b893-64c7f73a7e44 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:29.360265908 +0000 UTC m=+8.871325852 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aeef13dc-e06e-49c5-b893-64c7f73a7e44-insights-runtime-extractor-tls") pod "insights-runtime-extractor-smgf6" (UID: "aeef13dc-e06e-49c5-b893-64c7f73a7e44") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:16:25.360785 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:25.360498    5043 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:16:25.360785 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:25.360519    5043 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-748f8689f4-6lwjn: secret "image-registry-tls" not found
Oct 30 19:16:25.360785 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:25.360571    5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-registry-tls podName:9f302411-01be-438c-a30c-70c944afd630 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:29.36055335 +0000 UTC m=+8.871613307 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-registry-tls") pod "image-registry-748f8689f4-6lwjn" (UID: "9f302411-01be-438c-a30c-70c944afd630") : secret "image-registry-tls" not found
Oct 30 19:16:25.360785 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:25.360638    5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/247ff425-bed3-4297-b5c0-cc32b21bc179-service-ca-bundle podName:247ff425-bed3-4297-b5c0-cc32b21bc179 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:29.360628523 +0000 UTC m=+8.871688472 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/247ff425-bed3-4297-b5c0-cc32b21bc179-service-ca-bundle") pod "router-default-75866dc98c-4mnhf" (UID: "247ff425-bed3-4297-b5c0-cc32b21bc179") : configmap references non-existent config key: service-ca.crt
Oct 30 19:16:25.360785 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:25.360698    5043 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:16:25.360785 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:25.360707    5043 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78dc458d47-tqn8s: secret "image-registry-tls" not found
Oct 30 19:16:25.360785 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:25.360760    5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-registry-tls podName:a64211d0-1c48-41d2-b254-4ae84d5097d7 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:29.360725718 +0000 UTC m=+8.871785665 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-registry-tls") pod "image-registry-78dc458d47-tqn8s" (UID: "a64211d0-1c48-41d2-b254-4ae84d5097d7") : secret "image-registry-tls" not found
Oct 30 19:16:25.364019 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:25.363966    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a85cca66-a438-4117-9f50-977c7f528b7c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-lpg7w\" (UID: \"a85cca66-a438-4117-9f50-977c7f528b7c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-lpg7w"
Oct 30 19:16:25.378485 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:25.378456    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/06397fd4-3377-4eaa-8879-ebbb5943fbda-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2sdkw\" (UID: \"06397fd4-3377-4eaa-8879-ebbb5943fbda\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2sdkw"
Oct 30 19:16:25.505604 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:25.505565    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2sdkw"
Oct 30 19:16:25.506661 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:25.506636    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-lpg7w"
Oct 30 19:16:25.741797 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:25.741709    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gkn6q" event={"ID":"0269ffa6-ce06-43f3-be2a-58f6aa3e6964","Type":"ContainerStarted","Data":"1ba1e6c1bd943584f2c973a1c8cb51ef967e809a30dde6b0529ddc4a73d612c5"}
Oct 30 19:16:25.745303 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:25.745220    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p5nzf" event={"ID":"cbc4f7c7-bebd-4200-833e-1afbf3c2c81a","Type":"ContainerStarted","Data":"38ca2a418f757e051f986ddbd2861e503891a6507b8c87741eb077832081a1af"}
Oct 30 19:16:25.749661 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:25.749635    5043 generic.go:356] "Generic (PLEG): container finished" podID="68201777-2ec5-43b2-b587-81e585865d08" containerID="89510aef964a5db824e385074ed48b40781681da2cf2592625095bca48484db6" exitCode=0
Oct 30 19:16:25.749954 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:25.749905    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vd8tj" event={"ID":"68201777-2ec5-43b2-b587-81e585865d08","Type":"ContainerDied","Data":"89510aef964a5db824e385074ed48b40781681da2cf2592625095bca48484db6"}
Oct 30 19:16:25.750317 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:25.750303    5043 scope.go:122] "RemoveContainer" containerID="89510aef964a5db824e385074ed48b40781681da2cf2592625095bca48484db6"
Oct 30 19:16:25.761890 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:25.760851    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-fz9hf" event={"ID":"e5002691-8257-4dab-a88b-3c1b60e8e6f8","Type":"ContainerStarted","Data":"ad3aaa48836275356fc1d9cde537b18be4c7c0b71f9c4d9b4e4428dff4b2affc"}
Oct 30 19:16:25.843567 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:25.843473    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-lpg7w"]
Oct 30 19:16:25.844789 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:25.844096    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-2sdkw"]
Oct 30 19:16:25.845592 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:25.845568    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06397fd4_3377_4eaa_8879_ebbb5943fbda.slice/crio-ae6e2608fb8a8450e8048487c3b8a5e5d8910e766db6a4b86989d8483464ebcb WatchSource:0}: Error finding container ae6e2608fb8a8450e8048487c3b8a5e5d8910e766db6a4b86989d8483464ebcb: Status 404 returned error can't find the container with id ae6e2608fb8a8450e8048487c3b8a5e5d8910e766db6a4b86989d8483464ebcb
Oct 30 19:16:25.846114 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:25.846093    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda85cca66_a438_4117_9f50_977c7f528b7c.slice/crio-03e47ffd0dad897a20f062a62d0d95ce2023966d6664e4d0b2d620f3074cfc29 WatchSource:0}: Error finding container 03e47ffd0dad897a20f062a62d0d95ce2023966d6664e4d0b2d620f3074cfc29: Status 404 returned error can't find the container with id 03e47ffd0dad897a20f062a62d0d95ce2023966d6664e4d0b2d620f3074cfc29
Oct 30 19:16:26.588993 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:26.588954    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-cmw6w"
Oct 30 19:16:26.589434 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:26.589410    5043 scope.go:122] "RemoveContainer" containerID="981122b0f414c61dfba8d729d8a7e129d805b2486a8ad704c2cf06fb773ab2d4"
Oct 30 19:16:26.589661 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:26.589629    5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-cmw6w_openshift-console-operator(76e21916-0744-4499-a0cb-64a7a44a8285)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-cmw6w" podUID="76e21916-0744-4499-a0cb-64a7a44a8285"
Oct 30 19:16:26.767958 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:26.767921    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vd8tj" event={"ID":"68201777-2ec5-43b2-b587-81e585865d08","Type":"ContainerStarted","Data":"ae2fc782b548544a640db8295f4da90d7461686f36aed3133c547187ade9a56c"}
Oct 30 19:16:26.770516 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:26.770426    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-lpg7w" event={"ID":"a85cca66-a438-4117-9f50-977c7f528b7c","Type":"ContainerStarted","Data":"03e47ffd0dad897a20f062a62d0d95ce2023966d6664e4d0b2d620f3074cfc29"}
Oct 30 19:16:26.771989 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:26.771960    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2sdkw" event={"ID":"06397fd4-3377-4eaa-8879-ebbb5943fbda","Type":"ContainerStarted","Data":"ae6e2608fb8a8450e8048487c3b8a5e5d8910e766db6a4b86989d8483464ebcb"}
Oct 30 19:16:26.774856 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:26.774827    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gkn6q" event={"ID":"0269ffa6-ce06-43f3-be2a-58f6aa3e6964","Type":"ContainerStarted","Data":"f31ebfae86648320de8f45f0b016dcb0c66574635b5ec91e624ea4f5d9a186b1"}
Oct 30 19:16:26.775323 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:26.775307    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gkn6q"
Oct 30 19:16:26.777798 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:26.777775    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-p5nzf" event={"ID":"cbc4f7c7-bebd-4200-833e-1afbf3c2c81a","Type":"ContainerStarted","Data":"fd33550de45c98fa48d19ae66654881a7a6fcc019a362ac0ead1e436bf414c45"}
Oct 30 19:16:27.783719 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:27.783632    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-lpg7w" event={"ID":"a85cca66-a438-4117-9f50-977c7f528b7c","Type":"ContainerStarted","Data":"a03998378ae3a53aa1f74d6fea835718d41f1384bf916f65aee08b20904a0051"}
Oct 30 19:16:27.785761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:27.785704    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2sdkw" event={"ID":"06397fd4-3377-4eaa-8879-ebbb5943fbda","Type":"ContainerStarted","Data":"6114f043a633ea7be7cfbf027c746be0e95bf31921843507966d4a4a19a909e7"}
Oct 30 19:16:29.400559 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:29.400500    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aeef13dc-e06e-49c5-b893-64c7f73a7e44-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-smgf6\" (UID: \"aeef13dc-e06e-49c5-b893-64c7f73a7e44\") " pod="openshift-insights/insights-runtime-extractor-smgf6"
Oct 30 19:16:29.400559 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:29.400575    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/247ff425-bed3-4297-b5c0-cc32b21bc179-service-ca-bundle\") pod \"router-default-75866dc98c-4mnhf\" (UID: \"247ff425-bed3-4297-b5c0-cc32b21bc179\") " pod="openshift-ingress/router-default-75866dc98c-4mnhf"
Oct 30 19:16:29.401057 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:29.400595    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-registry-tls\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:29.401057 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:29.400630    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-registry-tls\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:29.401262 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:29.401238    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/247ff425-bed3-4297-b5c0-cc32b21bc179-service-ca-bundle\") pod \"router-default-75866dc98c-4mnhf\" (UID: \"247ff425-bed3-4297-b5c0-cc32b21bc179\") " pod="openshift-ingress/router-default-75866dc98c-4mnhf"
Oct 30 19:16:29.403184 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:29.403163    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aeef13dc-e06e-49c5-b893-64c7f73a7e44-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-smgf6\" (UID: \"aeef13dc-e06e-49c5-b893-64c7f73a7e44\") " pod="openshift-insights/insights-runtime-extractor-smgf6"
Oct 30 19:16:29.403285 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:29.403197    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-registry-tls\") pod \"image-registry-748f8689f4-6lwjn\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") " pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:29.403285 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:29.403278    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-75866dc98c-4mnhf"
Oct 30 19:16:29.403406 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:29.403368    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-registry-tls\") pod \"image-registry-78dc458d47-tqn8s\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") " pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:29.404152 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:29.404134    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:29.416269 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:29.416248    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:29.420471 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:29.420441    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-smgf6"
Oct 30 19:16:29.698825 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:29.698139    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-75866dc98c-4mnhf"]
Oct 30 19:16:29.702090 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:29.700835    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78dc458d47-tqn8s"]
Oct 30 19:16:29.703871 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:29.703840    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod247ff425_bed3_4297_b5c0_cc32b21bc179.slice/crio-c8eb292b5ca88df9f517d5406e1fd30a15e384ebeb593e5a241fb5a1f8ee4681 WatchSource:0}: Error finding container c8eb292b5ca88df9f517d5406e1fd30a15e384ebeb593e5a241fb5a1f8ee4681: Status 404 returned error can't find the container with id c8eb292b5ca88df9f517d5406e1fd30a15e384ebeb593e5a241fb5a1f8ee4681
Oct 30 19:16:29.706920 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:29.706757    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda64211d0_1c48_41d2_b254_4ae84d5097d7.slice/crio-463ed00d1868415901101bb15770724ee2f6db39a60d43f5c894ad5d9af3584c WatchSource:0}: Error finding container 463ed00d1868415901101bb15770724ee2f6db39a60d43f5c894ad5d9af3584c: Status 404 returned error can't find the container with id 463ed00d1868415901101bb15770724ee2f6db39a60d43f5c894ad5d9af3584c
Oct 30 19:16:29.729894 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:29.729871    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-smgf6"]
Oct 30 19:16:29.738850 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:29.736468    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-748f8689f4-6lwjn"]
Oct 30 19:16:29.739654 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:29.739633    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f302411_01be_438c_a30c_70c944afd630.slice/crio-247bbb32dc0ab7d347f4a2bba54493ab310f9e9a0a5529143bbc1bd929ef6a77 WatchSource:0}: Error finding container 247bbb32dc0ab7d347f4a2bba54493ab310f9e9a0a5529143bbc1bd929ef6a77: Status 404 returned error can't find the container with id 247bbb32dc0ab7d347f4a2bba54493ab310f9e9a0a5529143bbc1bd929ef6a77
Oct 30 19:16:29.797254 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:29.797225    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-748f8689f4-6lwjn" event={"ID":"9f302411-01be-438c-a30c-70c944afd630","Type":"ContainerStarted","Data":"247bbb32dc0ab7d347f4a2bba54493ab310f9e9a0a5529143bbc1bd929ef6a77"}
Oct 30 19:16:29.799773 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:29.799051    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-75866dc98c-4mnhf" event={"ID":"247ff425-bed3-4297-b5c0-cc32b21bc179","Type":"ContainerStarted","Data":"b0ac57d04a66f7bc01301905d733ec0ac9037338e7d8efb9a521c0c12eadffc3"}
Oct 30 19:16:29.799773 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:29.799090    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-75866dc98c-4mnhf" event={"ID":"247ff425-bed3-4297-b5c0-cc32b21bc179","Type":"ContainerStarted","Data":"c8eb292b5ca88df9f517d5406e1fd30a15e384ebeb593e5a241fb5a1f8ee4681"}
Oct 30 19:16:29.802584 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:29.802555    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78dc458d47-tqn8s" event={"ID":"a64211d0-1c48-41d2-b254-4ae84d5097d7","Type":"ContainerStarted","Data":"1f8ed7109b16da9cdfe1ac410e6e6c51704254271efb32b41c8ede21e78221f1"}
Oct 30 19:16:29.802711 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:29.802605    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78dc458d47-tqn8s" event={"ID":"a64211d0-1c48-41d2-b254-4ae84d5097d7","Type":"ContainerStarted","Data":"463ed00d1868415901101bb15770724ee2f6db39a60d43f5c894ad5d9af3584c"}
Oct 30 19:16:29.803378 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:29.803354    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:29.806321 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:29.806277    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-smgf6" event={"ID":"aeef13dc-e06e-49c5-b893-64c7f73a7e44","Type":"ContainerStarted","Data":"4e2f192e50544f5cc3c26bec9ec231b4be854cecbfac4b6905597673dba304b4"}
Oct 30 19:16:29.806478 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:29.806455    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-smgf6" event={"ID":"aeef13dc-e06e-49c5-b893-64c7f73a7e44","Type":"ContainerStarted","Data":"286b5d2b545ad115c7ce8745f07f9ba92df6f9b54aca5596a99542613f2e2857"}
Oct 30 19:16:30.280783 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:30.280727    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-n9zx2"
Oct 30 19:16:30.404310 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:30.404271    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-75866dc98c-4mnhf"
Oct 30 19:16:30.407371 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:30.407345    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-75866dc98c-4mnhf"
Oct 30 19:16:30.816042 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:30.815958    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-smgf6" event={"ID":"aeef13dc-e06e-49c5-b893-64c7f73a7e44","Type":"ContainerStarted","Data":"1dd5883db63ab47d90f21686365c0ff3416d083c1e16b932858f0a95920db804"}
Oct 30 19:16:30.817457 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:30.817420    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-748f8689f4-6lwjn" event={"ID":"9f302411-01be-438c-a30c-70c944afd630","Type":"ContainerStarted","Data":"8d18bc87003f8e30d5f4e70e8876ce3b391fea1dd1c19e63fbdaa5571766630b"}
Oct 30 19:16:31.169978 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:31.169902    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:31.757773 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:31.757716    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s8t94"
Oct 30 19:16:31.824176 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:31.824140    5043 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:16:32.185381 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:32.185287    5043 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-cmw6w"
Oct 30 19:16:32.185703 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:32.185690    5043 scope.go:122] "RemoveContainer" containerID="981122b0f414c61dfba8d729d8a7e129d805b2486a8ad704c2cf06fb773ab2d4"
Oct 30 19:16:32.830343 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:32.830304    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-smgf6" event={"ID":"aeef13dc-e06e-49c5-b893-64c7f73a7e44","Type":"ContainerStarted","Data":"05b09be087eb060739c45177613ad6bcae42217d13e7cc17de89dd9ced9f4272"}
Oct 30 19:16:32.832159 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:32.832143    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cmw6w_76e21916-0744-4499-a0cb-64a7a44a8285/console-operator/2.log"
Oct 30 19:16:32.832302 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:32.832279    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-cmw6w" event={"ID":"76e21916-0744-4499-a0cb-64a7a44a8285","Type":"ContainerStarted","Data":"51367f6cf13b1e63b55e4b519db921ccdb5824764df83dffa0c8edcc90118cc2"}
Oct 30 19:16:32.832646 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:32.832629    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-cmw6w"
Oct 30 19:16:32.837418 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:32.837395    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-cmw6w"
Oct 30 19:16:34.831215 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:34.831183    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-75866dc98c-4mnhf"
Oct 30 19:16:34.831655 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:34.831318    5043 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:16:34.832503 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:34.832483    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-75866dc98c-4mnhf"
Oct 30 19:16:36.305269 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.305237    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-tqp8q-5cb4z"]
Oct 30 19:16:36.312397 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.312361    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-tqp8q-5cb4z"
Oct 30 19:16:36.335392 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.335329    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-tqp8q-5cb4z"]
Oct 30 19:16:36.350461 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.350431    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bbdvn"]
Oct 30 19:16:36.362195 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.362162    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-78dc458d47-tqn8s"]
Oct 30 19:16:36.362411 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.362261    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bbdvn"
Oct 30 19:16:36.369690 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.369658    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bbdvn"]
Oct 30 19:16:36.369939 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.369921    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 30 19:16:36.372549 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.372528    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-nbgs8\""
Oct 30 19:16:36.429713 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.429686    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-wh4j2"]
Oct 30 19:16:36.433132 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.433111    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-f45b6f4dd-7rkpk"]
Oct 30 19:16:36.433251 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.433234    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-wh4j2"
Oct 30 19:16:36.437060 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.437020    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f45b6f4dd-7rkpk"
Oct 30 19:16:36.438434 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.438410    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 19:16:36.438434 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.438410    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 19:16:36.443323 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.443298    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-wh4j2"]
Oct 30 19:16:36.447250 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.447231    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f45b6f4dd-7rkpk"]
Oct 30 19:16:36.463784 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.463476    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llx9b\" (UniqueName: \"kubernetes.io/projected/030d88d7-44c3-4945-923e-1e57c2ffd18e-kube-api-access-llx9b\") pod \"periodic-gathering-tqp8q-5cb4z\" (UID: \"030d88d7-44c3-4945-923e-1e57c2ffd18e\") " pod="openshift-insights/periodic-gathering-tqp8q-5cb4z"
Oct 30 19:16:36.463784 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.463540    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/030d88d7-44c3-4945-923e-1e57c2ffd18e-trusted-ca-bundle\") pod \"periodic-gathering-tqp8q-5cb4z\" (UID: \"030d88d7-44c3-4945-923e-1e57c2ffd18e\") " pod="openshift-insights/periodic-gathering-tqp8q-5cb4z"
Oct 30 19:16:36.463784 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.463577    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/030d88d7-44c3-4945-923e-1e57c2ffd18e-service-ca-bundle\") pod \"periodic-gathering-tqp8q-5cb4z\" (UID: \"030d88d7-44c3-4945-923e-1e57c2ffd18e\") " pod="openshift-insights/periodic-gathering-tqp8q-5cb4z"
Oct 30 19:16:36.463784 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.463613    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/030d88d7-44c3-4945-923e-1e57c2ffd18e-archives-path\") pod \"periodic-gathering-tqp8q-5cb4z\" (UID: \"030d88d7-44c3-4945-923e-1e57c2ffd18e\") " pod="openshift-insights/periodic-gathering-tqp8q-5cb4z"
Oct 30 19:16:36.463784 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.463667    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/05d9b81e-3c5a-4590-ba33-6598855a9a5e-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-bbdvn\" (UID: \"05d9b81e-3c5a-4590-ba33-6598855a9a5e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bbdvn"
Oct 30 19:16:36.466993 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.466974    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-spth5\""
Oct 30 19:16:36.568239 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.568154    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/05d9b81e-3c5a-4590-ba33-6598855a9a5e-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-bbdvn\" (UID: \"05d9b81e-3c5a-4590-ba33-6598855a9a5e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bbdvn"
Oct 30 19:16:36.568239 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.568200    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/02190f18-2bb0-4f10-8a37-776da4e54aa3-installation-pull-secrets\") pod \"image-registry-f45b6f4dd-7rkpk\" (UID: \"02190f18-2bb0-4f10-8a37-776da4e54aa3\") " pod="openshift-image-registry/image-registry-f45b6f4dd-7rkpk"
Oct 30 19:16:36.568239 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.568223    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb415120-7b29-4553-ba70-cf2b3e00305e-tmp\") pod \"downloads-6dd789c8b7-wh4j2\" (UID: \"cb415120-7b29-4553-ba70-cf2b3e00305e\") " pod="openshift-console/downloads-6dd789c8b7-wh4j2"
Oct 30 19:16:36.568493 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.568248    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/02190f18-2bb0-4f10-8a37-776da4e54aa3-registry-tls\") pod \"image-registry-f45b6f4dd-7rkpk\" (UID: \"02190f18-2bb0-4f10-8a37-776da4e54aa3\") " pod="openshift-image-registry/image-registry-f45b6f4dd-7rkpk"
Oct 30 19:16:36.568493 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.568353    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/02190f18-2bb0-4f10-8a37-776da4e54aa3-ca-trust-extracted\") pod \"image-registry-f45b6f4dd-7rkpk\" (UID: \"02190f18-2bb0-4f10-8a37-776da4e54aa3\") " pod="openshift-image-registry/image-registry-f45b6f4dd-7rkpk"
Oct 30 19:16:36.568493 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.568388    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcwmt\" (UniqueName: \"kubernetes.io/projected/cb415120-7b29-4553-ba70-cf2b3e00305e-kube-api-access-xcwmt\") pod \"downloads-6dd789c8b7-wh4j2\" (UID: \"cb415120-7b29-4553-ba70-cf2b3e00305e\") " pod="openshift-console/downloads-6dd789c8b7-wh4j2"
Oct 30 19:16:36.568493 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.568469    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llx9b\" (UniqueName: \"kubernetes.io/projected/030d88d7-44c3-4945-923e-1e57c2ffd18e-kube-api-access-llx9b\") pod \"periodic-gathering-tqp8q-5cb4z\" (UID: \"030d88d7-44c3-4945-923e-1e57c2ffd18e\") " pod="openshift-insights/periodic-gathering-tqp8q-5cb4z"
Oct 30 19:16:36.568686 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.568511    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/02190f18-2bb0-4f10-8a37-776da4e54aa3-registry-certificates\") pod \"image-registry-f45b6f4dd-7rkpk\" (UID: \"02190f18-2bb0-4f10-8a37-776da4e54aa3\") " pod="openshift-image-registry/image-registry-f45b6f4dd-7rkpk"
Oct 30 19:16:36.568686 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.568527    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q529c\" (UniqueName: \"kubernetes.io/projected/02190f18-2bb0-4f10-8a37-776da4e54aa3-kube-api-access-q529c\") pod \"image-registry-f45b6f4dd-7rkpk\" (UID: \"02190f18-2bb0-4f10-8a37-776da4e54aa3\") " pod="openshift-image-registry/image-registry-f45b6f4dd-7rkpk"
Oct 30 19:16:36.568686 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.568620    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/02190f18-2bb0-4f10-8a37-776da4e54aa3-bound-sa-token\") pod \"image-registry-f45b6f4dd-7rkpk\" (UID: \"02190f18-2bb0-4f10-8a37-776da4e54aa3\") " pod="openshift-image-registry/image-registry-f45b6f4dd-7rkpk"
Oct 30 19:16:36.568686 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.568666    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/02190f18-2bb0-4f10-8a37-776da4e54aa3-image-registry-private-configuration\") pod \"image-registry-f45b6f4dd-7rkpk\" (UID: \"02190f18-2bb0-4f10-8a37-776da4e54aa3\") " pod="openshift-image-registry/image-registry-f45b6f4dd-7rkpk"
Oct 30 19:16:36.568869 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.568748    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/030d88d7-44c3-4945-923e-1e57c2ffd18e-trusted-ca-bundle\") pod \"periodic-gathering-tqp8q-5cb4z\" (UID: \"030d88d7-44c3-4945-923e-1e57c2ffd18e\") " pod="openshift-insights/periodic-gathering-tqp8q-5cb4z"
Oct 30 19:16:36.568869 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.568778    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/02190f18-2bb0-4f10-8a37-776da4e54aa3-trusted-ca\") pod \"image-registry-f45b6f4dd-7rkpk\" (UID: \"02190f18-2bb0-4f10-8a37-776da4e54aa3\") " pod="openshift-image-registry/image-registry-f45b6f4dd-7rkpk"
Oct 30 19:16:36.568869 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.568813    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/030d88d7-44c3-4945-923e-1e57c2ffd18e-service-ca-bundle\") pod \"periodic-gathering-tqp8q-5cb4z\" (UID: \"030d88d7-44c3-4945-923e-1e57c2ffd18e\") " pod="openshift-insights/periodic-gathering-tqp8q-5cb4z"
Oct 30 19:16:36.568869 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.568851    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/030d88d7-44c3-4945-923e-1e57c2ffd18e-archives-path\") pod \"periodic-gathering-tqp8q-5cb4z\" (UID: \"030d88d7-44c3-4945-923e-1e57c2ffd18e\") " pod="openshift-insights/periodic-gathering-tqp8q-5cb4z"
Oct 30 19:16:36.569863 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.569838    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/030d88d7-44c3-4945-923e-1e57c2ffd18e-service-ca-bundle\") pod \"periodic-gathering-tqp8q-5cb4z\" (UID: \"030d88d7-44c3-4945-923e-1e57c2ffd18e\") " pod="openshift-insights/periodic-gathering-tqp8q-5cb4z"
Oct 30 19:16:36.570000 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.569838    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/030d88d7-44c3-4945-923e-1e57c2ffd18e-trusted-ca-bundle\") pod \"periodic-gathering-tqp8q-5cb4z\" (UID: \"030d88d7-44c3-4945-923e-1e57c2ffd18e\") " pod="openshift-insights/periodic-gathering-tqp8q-5cb4z"
Oct 30 19:16:36.570069 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.570046    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/030d88d7-44c3-4945-923e-1e57c2ffd18e-archives-path\") pod \"periodic-gathering-tqp8q-5cb4z\" (UID: \"030d88d7-44c3-4945-923e-1e57c2ffd18e\") " pod="openshift-insights/periodic-gathering-tqp8q-5cb4z"
Oct 30 19:16:36.571286 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.571261    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/05d9b81e-3c5a-4590-ba33-6598855a9a5e-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-bbdvn\" (UID: \"05d9b81e-3c5a-4590-ba33-6598855a9a5e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bbdvn"
Oct 30 19:16:36.579009 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.578984    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llx9b\" (UniqueName: \"kubernetes.io/projected/030d88d7-44c3-4945-923e-1e57c2ffd18e-kube-api-access-llx9b\") pod \"periodic-gathering-tqp8q-5cb4z\" (UID: \"030d88d7-44c3-4945-923e-1e57c2ffd18e\") " pod="openshift-insights/periodic-gathering-tqp8q-5cb4z"
Oct 30 19:16:36.628192 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.628162    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-tqp8q-5cb4z"
Oct 30 19:16:36.670314 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.670278    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/02190f18-2bb0-4f10-8a37-776da4e54aa3-installation-pull-secrets\") pod \"image-registry-f45b6f4dd-7rkpk\" (UID: \"02190f18-2bb0-4f10-8a37-776da4e54aa3\") " pod="openshift-image-registry/image-registry-f45b6f4dd-7rkpk"
Oct 30 19:16:36.670314 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.670313    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb415120-7b29-4553-ba70-cf2b3e00305e-tmp\") pod \"downloads-6dd789c8b7-wh4j2\" (UID: \"cb415120-7b29-4553-ba70-cf2b3e00305e\") " pod="openshift-console/downloads-6dd789c8b7-wh4j2"
Oct 30 19:16:36.670538 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.670337    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/02190f18-2bb0-4f10-8a37-776da4e54aa3-registry-tls\") pod \"image-registry-f45b6f4dd-7rkpk\" (UID: \"02190f18-2bb0-4f10-8a37-776da4e54aa3\") " pod="openshift-image-registry/image-registry-f45b6f4dd-7rkpk"
Oct 30 19:16:36.670538 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.670355    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/02190f18-2bb0-4f10-8a37-776da4e54aa3-ca-trust-extracted\") pod \"image-registry-f45b6f4dd-7rkpk\" (UID: \"02190f18-2bb0-4f10-8a37-776da4e54aa3\") " pod="openshift-image-registry/image-registry-f45b6f4dd-7rkpk"
Oct 30 19:16:36.670538 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.670373    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcwmt\" (UniqueName: \"kubernetes.io/projected/cb415120-7b29-4553-ba70-cf2b3e00305e-kube-api-access-xcwmt\") pod \"downloads-6dd789c8b7-wh4j2\" (UID: \"cb415120-7b29-4553-ba70-cf2b3e00305e\") " pod="openshift-console/downloads-6dd789c8b7-wh4j2"
Oct 30 19:16:36.670538 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.670400    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/02190f18-2bb0-4f10-8a37-776da4e54aa3-registry-certificates\") pod \"image-registry-f45b6f4dd-7rkpk\" (UID: \"02190f18-2bb0-4f10-8a37-776da4e54aa3\") " pod="openshift-image-registry/image-registry-f45b6f4dd-7rkpk"
Oct 30 19:16:36.670538 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.670417    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q529c\" (UniqueName: \"kubernetes.io/projected/02190f18-2bb0-4f10-8a37-776da4e54aa3-kube-api-access-q529c\") pod \"image-registry-f45b6f4dd-7rkpk\" (UID: \"02190f18-2bb0-4f10-8a37-776da4e54aa3\") " pod="openshift-image-registry/image-registry-f45b6f4dd-7rkpk"
Oct 30 19:16:36.670538 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.670436    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/02190f18-2bb0-4f10-8a37-776da4e54aa3-bound-sa-token\") pod \"image-registry-f45b6f4dd-7rkpk\" (UID: \"02190f18-2bb0-4f10-8a37-776da4e54aa3\") " pod="openshift-image-registry/image-registry-f45b6f4dd-7rkpk"
Oct 30 19:16:36.670538 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.670456    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/02190f18-2bb0-4f10-8a37-776da4e54aa3-image-registry-private-configuration\") pod \"image-registry-f45b6f4dd-7rkpk\" (UID: \"02190f18-2bb0-4f10-8a37-776da4e54aa3\") " pod="openshift-image-registry/image-registry-f45b6f4dd-7rkpk"
Oct 30 19:16:36.670538 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.670473    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/02190f18-2bb0-4f10-8a37-776da4e54aa3-trusted-ca\") pod \"image-registry-f45b6f4dd-7rkpk\" (UID: \"02190f18-2bb0-4f10-8a37-776da4e54aa3\") " pod="openshift-image-registry/image-registry-f45b6f4dd-7rkpk"
Oct 30 19:16:36.671144 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.671118    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/02190f18-2bb0-4f10-8a37-776da4e54aa3-ca-trust-extracted\") pod \"image-registry-f45b6f4dd-7rkpk\" (UID: \"02190f18-2bb0-4f10-8a37-776da4e54aa3\") " pod="openshift-image-registry/image-registry-f45b6f4dd-7rkpk"
Oct 30 19:16:36.671235 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.671213    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb415120-7b29-4553-ba70-cf2b3e00305e-tmp\") pod \"downloads-6dd789c8b7-wh4j2\" (UID: \"cb415120-7b29-4553-ba70-cf2b3e00305e\") " pod="openshift-console/downloads-6dd789c8b7-wh4j2"
Oct 30 19:16:36.671282 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.671267    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/02190f18-2bb0-4f10-8a37-776da4e54aa3-trusted-ca\") pod \"image-registry-f45b6f4dd-7rkpk\" (UID: \"02190f18-2bb0-4f10-8a37-776da4e54aa3\") " pod="openshift-image-registry/image-registry-f45b6f4dd-7rkpk"
Oct 30 19:16:36.672101 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.672015    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/02190f18-2bb0-4f10-8a37-776da4e54aa3-registry-certificates\") pod \"image-registry-f45b6f4dd-7rkpk\" (UID: \"02190f18-2bb0-4f10-8a37-776da4e54aa3\") " pod="openshift-image-registry/image-registry-f45b6f4dd-7rkpk"
Oct 30 19:16:36.674143 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.674119    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/02190f18-2bb0-4f10-8a37-776da4e54aa3-installation-pull-secrets\") pod \"image-registry-f45b6f4dd-7rkpk\" (UID: \"02190f18-2bb0-4f10-8a37-776da4e54aa3\") " pod="openshift-image-registry/image-registry-f45b6f4dd-7rkpk"
Oct 30 19:16:36.674340 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.674312    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/02190f18-2bb0-4f10-8a37-776da4e54aa3-image-registry-private-configuration\") pod \"image-registry-f45b6f4dd-7rkpk\" (UID: \"02190f18-2bb0-4f10-8a37-776da4e54aa3\") " pod="openshift-image-registry/image-registry-f45b6f4dd-7rkpk"
Oct 30 19:16:36.674449 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.674433    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/02190f18-2bb0-4f10-8a37-776da4e54aa3-registry-tls\") pod \"image-registry-f45b6f4dd-7rkpk\" (UID: \"02190f18-2bb0-4f10-8a37-776da4e54aa3\") " pod="openshift-image-registry/image-registry-f45b6f4dd-7rkpk"
Oct 30 19:16:36.680380 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.680344    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q529c\" (UniqueName: \"kubernetes.io/projected/02190f18-2bb0-4f10-8a37-776da4e54aa3-kube-api-access-q529c\") pod \"image-registry-f45b6f4dd-7rkpk\" (UID: \"02190f18-2bb0-4f10-8a37-776da4e54aa3\") " pod="openshift-image-registry/image-registry-f45b6f4dd-7rkpk"
Oct 30 19:16:36.680498 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.680475    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcwmt\" (UniqueName: \"kubernetes.io/projected/cb415120-7b29-4553-ba70-cf2b3e00305e-kube-api-access-xcwmt\") pod \"downloads-6dd789c8b7-wh4j2\" (UID: \"cb415120-7b29-4553-ba70-cf2b3e00305e\") " pod="openshift-console/downloads-6dd789c8b7-wh4j2"
Oct 30 19:16:36.681114 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.681093    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/02190f18-2bb0-4f10-8a37-776da4e54aa3-bound-sa-token\") pod \"image-registry-f45b6f4dd-7rkpk\" (UID: \"02190f18-2bb0-4f10-8a37-776da4e54aa3\") " pod="openshift-image-registry/image-registry-f45b6f4dd-7rkpk"
Oct 30 19:16:36.683484 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.683459    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bbdvn"
Oct 30 19:16:36.748503 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.748212    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-wh4j2"
Oct 30 19:16:36.757719 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.757439    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f45b6f4dd-7rkpk"
Oct 30 19:16:36.934140 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.934117    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-tqp8q-5cb4z"]
Oct 30 19:16:36.936655 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:36.936624    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod030d88d7_44c3_4945_923e_1e57c2ffd18e.slice/crio-4128a3e88594cb1b3b1a35fee8fd68c9a0bc0625b8774db6006d00222181c6f2 WatchSource:0}: Error finding container 4128a3e88594cb1b3b1a35fee8fd68c9a0bc0625b8774db6006d00222181c6f2: Status 404 returned error can't find the container with id 4128a3e88594cb1b3b1a35fee8fd68c9a0bc0625b8774db6006d00222181c6f2
Oct 30 19:16:36.995927 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:36.995868    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bbdvn"]
Oct 30 19:16:37.003639 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:37.003614    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05d9b81e_3c5a_4590_ba33_6598855a9a5e.slice/crio-882f85d2a33f9db9c9b05afdd07d5c55160fc211fca25a484e84ae520c266476 WatchSource:0}: Error finding container 882f85d2a33f9db9c9b05afdd07d5c55160fc211fca25a484e84ae520c266476: Status 404 returned error can't find the container with id 882f85d2a33f9db9c9b05afdd07d5c55160fc211fca25a484e84ae520c266476
Oct 30 19:16:37.072931 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:37.072898    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-wh4j2"]
Oct 30 19:16:37.076538 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:37.076511    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f45b6f4dd-7rkpk"]
Oct 30 19:16:37.079619 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:37.079566    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb415120_7b29_4553_ba70_cf2b3e00305e.slice/crio-f1c94b8d38a7032f1f4ba453cd8dee72b7344a2c3d03dd1cf15cd8d647e427bc WatchSource:0}: Error finding container f1c94b8d38a7032f1f4ba453cd8dee72b7344a2c3d03dd1cf15cd8d647e427bc: Status 404 returned error can't find the container with id f1c94b8d38a7032f1f4ba453cd8dee72b7344a2c3d03dd1cf15cd8d647e427bc
Oct 30 19:16:37.792397 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:37.792365    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gkn6q"
Oct 30 19:16:37.862867 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:37.862787    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-wh4j2" event={"ID":"cb415120-7b29-4553-ba70-cf2b3e00305e","Type":"ContainerStarted","Data":"f1c94b8d38a7032f1f4ba453cd8dee72b7344a2c3d03dd1cf15cd8d647e427bc"}
Oct 30 19:16:37.866212 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:37.866178    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-tqp8q-5cb4z" event={"ID":"030d88d7-44c3-4945-923e-1e57c2ffd18e","Type":"ContainerStarted","Data":"2318bcf1dcdff54f185aa4d3631ad9e599556918079a229669345c5111797f8c"}
Oct 30 19:16:37.866349 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:37.866232    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-tqp8q-5cb4z" event={"ID":"030d88d7-44c3-4945-923e-1e57c2ffd18e","Type":"ContainerStarted","Data":"4128a3e88594cb1b3b1a35fee8fd68c9a0bc0625b8774db6006d00222181c6f2"}
Oct 30 19:16:37.868307 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:37.868277    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f45b6f4dd-7rkpk" event={"ID":"02190f18-2bb0-4f10-8a37-776da4e54aa3","Type":"ContainerStarted","Data":"b59b85f1b718ce8a567a9ead5896d6aac9e528bc4dea4a1f623ce54c589af963"}
Oct 30 19:16:37.868436 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:37.868334    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-f45b6f4dd-7rkpk"
Oct 30 19:16:37.868436 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:37.868364    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f45b6f4dd-7rkpk" event={"ID":"02190f18-2bb0-4f10-8a37-776da4e54aa3","Type":"ContainerStarted","Data":"5359e105ca9a9780b3278b311a70d73ffa6774b6e81c4da4a5e8ad44a1832c88"}
Oct 30 19:16:37.869965 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:37.869938    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bbdvn" event={"ID":"05d9b81e-3c5a-4590-ba33-6598855a9a5e","Type":"ContainerStarted","Data":"882f85d2a33f9db9c9b05afdd07d5c55160fc211fca25a484e84ae520c266476"}
Oct 30 19:16:37.890405 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:37.890343    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-tqp8q-5cb4z" podStartSLOduration=1.8903261169999999 podStartE2EDuration="1.890326117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:37.888913746 +0000 UTC m=+17.399973719" watchObservedRunningTime="2026-10-30 19:16:37.890326117 +0000 UTC m=+17.401386084"
Oct 30 19:16:37.912788 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:37.912368    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-f45b6f4dd-7rkpk" podStartSLOduration=1.9123460749999999 podStartE2EDuration="1.912346075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:37.910972439 +0000 UTC m=+17.422032408" watchObservedRunningTime="2026-10-30 19:16:37.912346075 +0000 UTC m=+17.423406045"
Oct 30 19:16:38.215001 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:38.214976    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gkn6q_0269ffa6-ce06-43f3-be2a-58f6aa3e6964/dns/0.log"
Oct 30 19:16:38.399170 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:38.399110    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gkn6q_0269ffa6-ce06-43f3-be2a-58f6aa3e6964/kube-rbac-proxy/0.log"
Oct 30 19:16:38.876642 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:38.876601    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bbdvn" event={"ID":"05d9b81e-3c5a-4590-ba33-6598855a9a5e","Type":"ContainerStarted","Data":"308a569bb5db7b0f43a069e61a97ef70244125b9a783d65951580c4a7576390b"}
Oct 30 19:16:38.877158 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:38.877076    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bbdvn"
Oct 30 19:16:38.882663 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:38.882626    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bbdvn"
Oct 30 19:16:38.893489 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:38.893087    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bbdvn" podStartSLOduration=1.736096444 podStartE2EDuration="2.893075191s" totalImagesPullingTime="1.156978747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:38.892557113 +0000 UTC m=+18.403617078" watchObservedRunningTime="2026-10-30 19:16:38.893075191 +0000 UTC m=+18.404135153"
Oct 30 19:16:38.998547 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:38.998518    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xjjqh_36ab52a2-65e2-4af6-a872-9052e12714d1/dns-node-resolver/0.log"
Oct 30 19:16:39.198785 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:39.198692    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-748f8689f4-6lwjn_9f302411-01be-438c-a30c-70c944afd630/registry/0.log"
Oct 30 19:16:39.399053 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:39.399022    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-78dc458d47-tqn8s_a64211d0-1c48-41d2-b254-4ae84d5097d7/registry/0.log"
Oct 30 19:16:39.599406 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:39.599372    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-f45b6f4dd-7rkpk_02190f18-2bb0-4f10-8a37-776da4e54aa3/registry/0.log"
Oct 30 19:16:39.650450 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:39.650417    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lqmqr"
Oct 30 19:16:40.197908 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:40.197878    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zj8d6_f1ea27b2-80df-404c-ac39-460f495957b2/node-ca/0.log"
Oct 30 19:16:40.399348 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:40.399313    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-75866dc98c-4mnhf_247ff425-bed3-4297-b5c0-cc32b21bc179/router/0.log"
Oct 30 19:16:40.798659 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:40.798635    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9jkcb_765d8d9d-b89f-419f-b237-3f8034db1bed/serve-healthcheck-canary/0.log"
Oct 30 19:16:43.558900 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:43.556880    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nrt89"]
Oct 30 19:16:43.600931 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:43.599601    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nrt89"
Oct 30 19:16:43.650179 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:43.650083    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 19:16:43.650347 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:43.650269    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 19:16:43.650347 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:43.650283    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 19:16:43.650347 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:43.650337    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 19:16:43.650480 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:43.650355    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bc6qp\""
Oct 30 19:16:43.745995 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:43.745961    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0359a57a-4c15-4eba-bc01-7d5b5d3d97bf-root\") pod \"node-exporter-nrt89\" (UID: \"0359a57a-4c15-4eba-bc01-7d5b5d3d97bf\") " pod="openshift-monitoring/node-exporter-nrt89"
Oct 30 19:16:43.747515 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:43.746010    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0359a57a-4c15-4eba-bc01-7d5b5d3d97bf-node-exporter-tls\") pod \"node-exporter-nrt89\" (UID: \"0359a57a-4c15-4eba-bc01-7d5b5d3d97bf\") " pod="openshift-monitoring/node-exporter-nrt89"
Oct 30 19:16:43.747515 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:43.746112    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0359a57a-4c15-4eba-bc01-7d5b5d3d97bf-metrics-client-ca\") pod \"node-exporter-nrt89\" (UID: \"0359a57a-4c15-4eba-bc01-7d5b5d3d97bf\") " pod="openshift-monitoring/node-exporter-nrt89"
Oct 30 19:16:43.747515 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:43.746175    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0359a57a-4c15-4eba-bc01-7d5b5d3d97bf-node-exporter-accelerators-collector-config\") pod \"node-exporter-nrt89\" (UID: \"0359a57a-4c15-4eba-bc01-7d5b5d3d97bf\") " pod="openshift-monitoring/node-exporter-nrt89"
Oct 30 19:16:43.747515 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:43.746224    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0359a57a-4c15-4eba-bc01-7d5b5d3d97bf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nrt89\" (UID: \"0359a57a-4c15-4eba-bc01-7d5b5d3d97bf\") " pod="openshift-monitoring/node-exporter-nrt89"
Oct 30 19:16:43.747515 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:43.746257    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0359a57a-4c15-4eba-bc01-7d5b5d3d97bf-node-exporter-textfile\") pod \"node-exporter-nrt89\" (UID: \"0359a57a-4c15-4eba-bc01-7d5b5d3d97bf\") " pod="openshift-monitoring/node-exporter-nrt89"
Oct 30 19:16:43.747515 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:43.746292    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0359a57a-4c15-4eba-bc01-7d5b5d3d97bf-sys\") pod \"node-exporter-nrt89\" (UID: \"0359a57a-4c15-4eba-bc01-7d5b5d3d97bf\") " pod="openshift-monitoring/node-exporter-nrt89"
Oct 30 19:16:43.747515 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:43.746316    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0359a57a-4c15-4eba-bc01-7d5b5d3d97bf-node-exporter-wtmp\") pod \"node-exporter-nrt89\" (UID: \"0359a57a-4c15-4eba-bc01-7d5b5d3d97bf\") " pod="openshift-monitoring/node-exporter-nrt89"
Oct 30 19:16:43.747515 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:43.746347    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjtxj\" (UniqueName: \"kubernetes.io/projected/0359a57a-4c15-4eba-bc01-7d5b5d3d97bf-kube-api-access-cjtxj\") pod \"node-exporter-nrt89\" (UID: \"0359a57a-4c15-4eba-bc01-7d5b5d3d97bf\") " pod="openshift-monitoring/node-exporter-nrt89"
Oct 30 19:16:43.847377 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:43.847293    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0359a57a-4c15-4eba-bc01-7d5b5d3d97bf-sys\") pod \"node-exporter-nrt89\" (UID: \"0359a57a-4c15-4eba-bc01-7d5b5d3d97bf\") " pod="openshift-monitoring/node-exporter-nrt89"
Oct 30 19:16:43.847377 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:43.847329    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0359a57a-4c15-4eba-bc01-7d5b5d3d97bf-node-exporter-wtmp\") pod \"node-exporter-nrt89\" (UID: \"0359a57a-4c15-4eba-bc01-7d5b5d3d97bf\") " pod="openshift-monitoring/node-exporter-nrt89"
Oct 30 19:16:43.847377 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:43.847360    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjtxj\" (UniqueName: \"kubernetes.io/projected/0359a57a-4c15-4eba-bc01-7d5b5d3d97bf-kube-api-access-cjtxj\") pod \"node-exporter-nrt89\" (UID: \"0359a57a-4c15-4eba-bc01-7d5b5d3d97bf\") " pod="openshift-monitoring/node-exporter-nrt89"
Oct 30 19:16:43.847611 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:43.847415    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0359a57a-4c15-4eba-bc01-7d5b5d3d97bf-root\") pod \"node-exporter-nrt89\" (UID: \"0359a57a-4c15-4eba-bc01-7d5b5d3d97bf\") " pod="openshift-monitoring/node-exporter-nrt89"
Oct 30 19:16:43.847611 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:43.847417    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0359a57a-4c15-4eba-bc01-7d5b5d3d97bf-sys\") pod \"node-exporter-nrt89\" (UID: \"0359a57a-4c15-4eba-bc01-7d5b5d3d97bf\") " pod="openshift-monitoring/node-exporter-nrt89"
Oct 30 19:16:43.847611 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:43.847444    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0359a57a-4c15-4eba-bc01-7d5b5d3d97bf-node-exporter-tls\") pod \"node-exporter-nrt89\" (UID: \"0359a57a-4c15-4eba-bc01-7d5b5d3d97bf\") " pod="openshift-monitoring/node-exporter-nrt89"
Oct 30 19:16:43.847611 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:43.847467    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0359a57a-4c15-4eba-bc01-7d5b5d3d97bf-metrics-client-ca\") pod \"node-exporter-nrt89\" (UID: \"0359a57a-4c15-4eba-bc01-7d5b5d3d97bf\") " pod="openshift-monitoring/node-exporter-nrt89"
Oct 30 19:16:43.847611 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:43.847489    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0359a57a-4c15-4eba-bc01-7d5b5d3d97bf-root\") pod \"node-exporter-nrt89\" (UID: \"0359a57a-4c15-4eba-bc01-7d5b5d3d97bf\") " pod="openshift-monitoring/node-exporter-nrt89"
Oct 30 19:16:43.847611 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:43.847509    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0359a57a-4c15-4eba-bc01-7d5b5d3d97bf-node-exporter-wtmp\") pod \"node-exporter-nrt89\" (UID: \"0359a57a-4c15-4eba-bc01-7d5b5d3d97bf\") " pod="openshift-monitoring/node-exporter-nrt89"
Oct 30 19:16:43.847611 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:43.847512    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0359a57a-4c15-4eba-bc01-7d5b5d3d97bf-node-exporter-accelerators-collector-config\") pod \"node-exporter-nrt89\" (UID: \"0359a57a-4c15-4eba-bc01-7d5b5d3d97bf\") " pod="openshift-monitoring/node-exporter-nrt89"
Oct 30 19:16:43.847611 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:43.847580    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0359a57a-4c15-4eba-bc01-7d5b5d3d97bf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nrt89\" (UID: \"0359a57a-4c15-4eba-bc01-7d5b5d3d97bf\") " pod="openshift-monitoring/node-exporter-nrt89"
Oct 30 19:16:43.847611 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:43.847605    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0359a57a-4c15-4eba-bc01-7d5b5d3d97bf-node-exporter-textfile\") pod \"node-exporter-nrt89\" (UID: \"0359a57a-4c15-4eba-bc01-7d5b5d3d97bf\") " pod="openshift-monitoring/node-exporter-nrt89"
Oct 30 19:16:43.847980 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:43.847931    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0359a57a-4c15-4eba-bc01-7d5b5d3d97bf-node-exporter-textfile\") pod \"node-exporter-nrt89\" (UID: \"0359a57a-4c15-4eba-bc01-7d5b5d3d97bf\") " pod="openshift-monitoring/node-exporter-nrt89"
Oct 30 19:16:43.848182 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:43.848155    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0359a57a-4c15-4eba-bc01-7d5b5d3d97bf-node-exporter-accelerators-collector-config\") pod \"node-exporter-nrt89\" (UID: \"0359a57a-4c15-4eba-bc01-7d5b5d3d97bf\") " pod="openshift-monitoring/node-exporter-nrt89"
Oct 30 19:16:43.848250 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:43.848216    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0359a57a-4c15-4eba-bc01-7d5b5d3d97bf-metrics-client-ca\") pod \"node-exporter-nrt89\" (UID: \"0359a57a-4c15-4eba-bc01-7d5b5d3d97bf\") " pod="openshift-monitoring/node-exporter-nrt89"
Oct 30 19:16:43.850394 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:43.850355    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0359a57a-4c15-4eba-bc01-7d5b5d3d97bf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nrt89\" (UID: \"0359a57a-4c15-4eba-bc01-7d5b5d3d97bf\") " pod="openshift-monitoring/node-exporter-nrt89"
Oct 30 19:16:43.850394 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:43.850381    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0359a57a-4c15-4eba-bc01-7d5b5d3d97bf-node-exporter-tls\") pod \"node-exporter-nrt89\" (UID: \"0359a57a-4c15-4eba-bc01-7d5b5d3d97bf\") " pod="openshift-monitoring/node-exporter-nrt89"
Oct 30 19:16:43.856046 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:43.856021    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjtxj\" (UniqueName: \"kubernetes.io/projected/0359a57a-4c15-4eba-bc01-7d5b5d3d97bf-kube-api-access-cjtxj\") pod \"node-exporter-nrt89\" (UID: \"0359a57a-4c15-4eba-bc01-7d5b5d3d97bf\") " pod="openshift-monitoring/node-exporter-nrt89"
Oct 30 19:16:43.969607 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:43.969572    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nrt89"
Oct 30 19:16:44.011262 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:16:44.011219    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0359a57a_4c15_4eba_bc01_7d5b5d3d97bf.slice/crio-6bfa2a61d51b548e020f244f849da3238d15dafc0cddf682efafa8d32cf8ea46 WatchSource:0}: Error finding container 6bfa2a61d51b548e020f244f849da3238d15dafc0cddf682efafa8d32cf8ea46: Status 404 returned error can't find the container with id 6bfa2a61d51b548e020f244f849da3238d15dafc0cddf682efafa8d32cf8ea46
Oct 30 19:16:44.907704 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:44.907669    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nrt89" event={"ID":"0359a57a-4c15-4eba-bc01-7d5b5d3d97bf","Type":"ContainerStarted","Data":"6bfa2a61d51b548e020f244f849da3238d15dafc0cddf682efafa8d32cf8ea46"}
Oct 30 19:16:45.914714 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:45.914634    5043 generic.go:356] "Generic (PLEG): container finished" podID="0359a57a-4c15-4eba-bc01-7d5b5d3d97bf" containerID="2b90fd0c7a99b812808b5772ce96076b10b78af45789d144ebcded50a2d1c6c7" exitCode=0
Oct 30 19:16:45.914714 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:45.914691    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nrt89" event={"ID":"0359a57a-4c15-4eba-bc01-7d5b5d3d97bf","Type":"ContainerDied","Data":"2b90fd0c7a99b812808b5772ce96076b10b78af45789d144ebcded50a2d1c6c7"}
Oct 30 19:16:46.369546 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:46.369520    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:16:46.922929 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:46.922881    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nrt89" event={"ID":"0359a57a-4c15-4eba-bc01-7d5b5d3d97bf","Type":"ContainerStarted","Data":"1b840af68bf3c80acf78f1dcb5f43389d80892ba4da024cee42cf15a11248562"}
Oct 30 19:16:46.923381 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:46.922950    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nrt89" event={"ID":"0359a57a-4c15-4eba-bc01-7d5b5d3d97bf","Type":"ContainerStarted","Data":"b526707f6c5a59b815e4b36983e3c3c7dbc5274e959015fdd24f11970547f06d"}
Oct 30 19:16:46.943712 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:46.943657    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nrt89" podStartSLOduration=2.279469224 podStartE2EDuration="3.943641059s" totalImagesPullingTime="1.564031249s" totalInitContainerRuntime="100.140586ms" isStatefulPod=true podCreationTimestamp="2026-10-30 19:16:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:46.942432462 +0000 UTC m=+26.453492428" watchObservedRunningTime="2026-10-30 19:16:46.943641059 +0000 UTC m=+26.454701017"
Oct 30 19:16:50.284887 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:50.284850    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-n9zx2"
Oct 30 19:16:52.838257 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:52.838213    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:16:53.958855 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:53.958771    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-wh4j2" event={"ID":"cb415120-7b29-4553-ba70-cf2b3e00305e","Type":"ContainerStarted","Data":"99e0cecc6a9572cefd096cc4da6dde5394457f9b6de47b8042ab3350950771f6"}
Oct 30 19:16:53.959298 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:53.958946    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-wh4j2"
Oct 30 19:16:53.960892 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:53.960862    5043 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-wh4j2 container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.30:8080/\": dial tcp 10.132.0.30:8080: connect: connection refused" start-of-body=
Oct 30 19:16:53.961006 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:53.960912    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-wh4j2" podUID="cb415120-7b29-4553-ba70-cf2b3e00305e" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.30:8080/\": dial tcp 10.132.0.30:8080: connect: connection refused"
Oct 30 19:16:53.978097 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:53.978045    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-wh4j2" podStartSLOduration=1.508112007 podStartE2EDuration="17.978030467s" totalImagesPullingTime="16.46991846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:53.976668623 +0000 UTC m=+33.487728588" watchObservedRunningTime="2026-10-30 19:16:53.978030467 +0000 UTC m=+33.489090423"
Oct 30 19:16:54.972092 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:54.972062    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-wh4j2"
Oct 30 19:16:54.980855 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:16:54.980790    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:16:58.893844 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:58.893569    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-f45b6f4dd-7rkpk"
Oct 30 19:16:58.952314 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:16:58.952264    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-748f8689f4-6lwjn"]
Oct 30 19:17:01.390688 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:01.390565    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-78dc458d47-tqn8s" podUID="a64211d0-1c48-41d2-b254-4ae84d5097d7" containerName="registry" containerID="cri-o://1f8ed7109b16da9cdfe1ac410e6e6c51704254271efb32b41c8ede21e78221f1" gracePeriod=30
Oct 30 19:17:01.710905 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:01.710875    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:17:01.816469 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:01.816429    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a64211d0-1c48-41d2-b254-4ae84d5097d7-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a64211d0-1c48-41d2-b254-4ae84d5097d7-installation-pull-secrets\") pod \"a64211d0-1c48-41d2-b254-4ae84d5097d7\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") "
Oct 30 19:17:01.816681 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:01.816540    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a64211d0-1c48-41d2-b254-4ae84d5097d7-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a64211d0-1c48-41d2-b254-4ae84d5097d7-ca-trust-extracted\") pod \"a64211d0-1c48-41d2-b254-4ae84d5097d7\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") "
Oct 30 19:17:01.816681 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:01.816590    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a64211d0-1c48-41d2-b254-4ae84d5097d7-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a64211d0-1c48-41d2-b254-4ae84d5097d7-trusted-ca\") pod \"a64211d0-1c48-41d2-b254-4ae84d5097d7\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") "
Oct 30 19:17:01.816681 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:01.816637    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a64211d0-1c48-41d2-b254-4ae84d5097d7-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a64211d0-1c48-41d2-b254-4ae84d5097d7-registry-certificates\") pod \"a64211d0-1c48-41d2-b254-4ae84d5097d7\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") "
Oct 30 19:17:01.816681 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:01.816663    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-kube-api-access-6x795\" (UniqueName: \"kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-kube-api-access-6x795\") pod \"a64211d0-1c48-41d2-b254-4ae84d5097d7\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") "
Oct 30 19:17:01.816681 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:01.816679    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-registry-tls\" (UniqueName: \"kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-registry-tls\") pod \"a64211d0-1c48-41d2-b254-4ae84d5097d7\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") "
Oct 30 19:17:01.816977 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:01.816710    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-bound-sa-token\") pod \"a64211d0-1c48-41d2-b254-4ae84d5097d7\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") "
Oct 30 19:17:01.816977 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:01.816768    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a64211d0-1c48-41d2-b254-4ae84d5097d7-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a64211d0-1c48-41d2-b254-4ae84d5097d7-image-registry-private-configuration\") pod \"a64211d0-1c48-41d2-b254-4ae84d5097d7\" (UID: \"a64211d0-1c48-41d2-b254-4ae84d5097d7\") "
Oct 30 19:17:01.818063 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:01.818001    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a64211d0-1c48-41d2-b254-4ae84d5097d7-trusted-ca" pod "a64211d0-1c48-41d2-b254-4ae84d5097d7" (UID: "a64211d0-1c48-41d2-b254-4ae84d5097d7"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:01.818400 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:01.818362    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a64211d0-1c48-41d2-b254-4ae84d5097d7-registry-certificates" pod "a64211d0-1c48-41d2-b254-4ae84d5097d7" (UID: "a64211d0-1c48-41d2-b254-4ae84d5097d7"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:01.820202 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:01.820157    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-kube-api-access-6x795" pod "a64211d0-1c48-41d2-b254-4ae84d5097d7" (UID: "a64211d0-1c48-41d2-b254-4ae84d5097d7"). InnerVolumeSpecName "kube-api-access-6x795". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:01.820443 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:01.820417    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-registry-tls" pod "a64211d0-1c48-41d2-b254-4ae84d5097d7" (UID: "a64211d0-1c48-41d2-b254-4ae84d5097d7"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:01.821560 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:01.821512    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a64211d0-1c48-41d2-b254-4ae84d5097d7-installation-pull-secrets" pod "a64211d0-1c48-41d2-b254-4ae84d5097d7" (UID: "a64211d0-1c48-41d2-b254-4ae84d5097d7"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:01.821666 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:01.821606    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a64211d0-1c48-41d2-b254-4ae84d5097d7-image-registry-private-configuration" pod "a64211d0-1c48-41d2-b254-4ae84d5097d7" (UID: "a64211d0-1c48-41d2-b254-4ae84d5097d7"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:01.821907 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:01.821883    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-bound-sa-token" pod "a64211d0-1c48-41d2-b254-4ae84d5097d7" (UID: "a64211d0-1c48-41d2-b254-4ae84d5097d7"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:01.826249 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:01.826221    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a64211d0-1c48-41d2-b254-4ae84d5097d7-ca-trust-extracted" pod "a64211d0-1c48-41d2-b254-4ae84d5097d7" (UID: "a64211d0-1c48-41d2-b254-4ae84d5097d7"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:01.918311 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:01.918270    5043 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a64211d0-1c48-41d2-b254-4ae84d5097d7-installation-pull-secrets\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:01.918311 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:01.918307    5043 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a64211d0-1c48-41d2-b254-4ae84d5097d7-ca-trust-extracted\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:01.918542 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:01.918328    5043 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a64211d0-1c48-41d2-b254-4ae84d5097d7-trusted-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:01.918542 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:01.918343    5043 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a64211d0-1c48-41d2-b254-4ae84d5097d7-registry-certificates\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:01.918542 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:01.918358    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x795\" (UniqueName: \"kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-kube-api-access-6x795\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:01.918542 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:01.918375    5043 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-registry-tls\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:01.918542 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:01.918391    5043 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a64211d0-1c48-41d2-b254-4ae84d5097d7-bound-sa-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:01.918542 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:01.918406    5043 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a64211d0-1c48-41d2-b254-4ae84d5097d7-image-registry-private-configuration\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:02.020217 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:02.020181    5043 generic.go:356] "Generic (PLEG): container finished" podID="a64211d0-1c48-41d2-b254-4ae84d5097d7" containerID="1f8ed7109b16da9cdfe1ac410e6e6c51704254271efb32b41c8ede21e78221f1" exitCode=0
Oct 30 19:17:02.020402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:02.020235    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78dc458d47-tqn8s"
Oct 30 19:17:02.020402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:02.020243    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78dc458d47-tqn8s" event={"ID":"a64211d0-1c48-41d2-b254-4ae84d5097d7","Type":"ContainerDied","Data":"1f8ed7109b16da9cdfe1ac410e6e6c51704254271efb32b41c8ede21e78221f1"}
Oct 30 19:17:02.020402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:02.020347    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78dc458d47-tqn8s" event={"ID":"a64211d0-1c48-41d2-b254-4ae84d5097d7","Type":"ContainerDied","Data":"463ed00d1868415901101bb15770724ee2f6db39a60d43f5c894ad5d9af3584c"}
Oct 30 19:17:02.020402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:02.020397    5043 scope.go:122] "RemoveContainer" containerID="1f8ed7109b16da9cdfe1ac410e6e6c51704254271efb32b41c8ede21e78221f1"
Oct 30 19:17:02.031903 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:02.031881    5043 scope.go:122] "RemoveContainer" containerID="1f8ed7109b16da9cdfe1ac410e6e6c51704254271efb32b41c8ede21e78221f1"
Oct 30 19:17:02.032284 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:17:02.032253    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f8ed7109b16da9cdfe1ac410e6e6c51704254271efb32b41c8ede21e78221f1\": container with ID starting with 1f8ed7109b16da9cdfe1ac410e6e6c51704254271efb32b41c8ede21e78221f1 not found: ID does not exist" containerID="1f8ed7109b16da9cdfe1ac410e6e6c51704254271efb32b41c8ede21e78221f1"
Oct 30 19:17:02.032358 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:02.032300    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f8ed7109b16da9cdfe1ac410e6e6c51704254271efb32b41c8ede21e78221f1"} err="failed to get container status \"1f8ed7109b16da9cdfe1ac410e6e6c51704254271efb32b41c8ede21e78221f1\": rpc error: code = NotFound desc = could not find container \"1f8ed7109b16da9cdfe1ac410e6e6c51704254271efb32b41c8ede21e78221f1\": container with ID starting with 1f8ed7109b16da9cdfe1ac410e6e6c51704254271efb32b41c8ede21e78221f1 not found: ID does not exist"
Oct 30 19:17:02.051970 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:02.051944    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-78dc458d47-tqn8s"]
Oct 30 19:17:02.065542 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:02.065500    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-78dc458d47-tqn8s"]
Oct 30 19:17:02.643385 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:02.643354    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a64211d0-1c48-41d2-b254-4ae84d5097d7" path="/var/lib/kubelet/pods/a64211d0-1c48-41d2-b254-4ae84d5097d7/volumes"
Oct 30 19:17:08.779060 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:08.779020    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-2lcwk-gjv8f"]
Oct 30 19:17:08.779598 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:08.779280    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-2lcwk-gjv8f" podUID="5338b461-6b7c-474b-8458-d1b53775ee0f" containerName="insights-gathering" containerID="cri-o://6081dbd078e020eb140d07eec8006a13c5f61e5409c51c7e49470b00cd9133f6" gracePeriod=30
Oct 30 19:17:09.055090 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:09.055055    5043 generic.go:356] "Generic (PLEG): container finished" podID="5338b461-6b7c-474b-8458-d1b53775ee0f" containerID="6081dbd078e020eb140d07eec8006a13c5f61e5409c51c7e49470b00cd9133f6" exitCode=2
Oct 30 19:17:09.055263 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:09.055135    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-2lcwk-gjv8f" event={"ID":"5338b461-6b7c-474b-8458-d1b53775ee0f","Type":"ContainerDied","Data":"6081dbd078e020eb140d07eec8006a13c5f61e5409c51c7e49470b00cd9133f6"}
Oct 30 19:17:09.086178 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:09.086149    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-2lcwk-gjv8f"
Oct 30 19:17:09.176893 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:09.176854    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5338b461-6b7c-474b-8458-d1b53775ee0f-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5338b461-6b7c-474b-8458-d1b53775ee0f-archives-path\") pod \"5338b461-6b7c-474b-8458-d1b53775ee0f\" (UID: \"5338b461-6b7c-474b-8458-d1b53775ee0f\") "
Oct 30 19:17:09.177078 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:09.176927    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5338b461-6b7c-474b-8458-d1b53775ee0f-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5338b461-6b7c-474b-8458-d1b53775ee0f-trusted-ca-bundle\") pod \"5338b461-6b7c-474b-8458-d1b53775ee0f\" (UID: \"5338b461-6b7c-474b-8458-d1b53775ee0f\") "
Oct 30 19:17:09.177078 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:09.176976    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5338b461-6b7c-474b-8458-d1b53775ee0f-kube-api-access-4xrrr\" (UniqueName: \"kubernetes.io/projected/5338b461-6b7c-474b-8458-d1b53775ee0f-kube-api-access-4xrrr\") pod \"5338b461-6b7c-474b-8458-d1b53775ee0f\" (UID: \"5338b461-6b7c-474b-8458-d1b53775ee0f\") "
Oct 30 19:17:09.177078 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:09.177058    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5338b461-6b7c-474b-8458-d1b53775ee0f-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5338b461-6b7c-474b-8458-d1b53775ee0f-service-ca-bundle\") pod \"5338b461-6b7c-474b-8458-d1b53775ee0f\" (UID: \"5338b461-6b7c-474b-8458-d1b53775ee0f\") "
Oct 30 19:17:09.177336 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:09.177313    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5338b461-6b7c-474b-8458-d1b53775ee0f-archives-path" pod "5338b461-6b7c-474b-8458-d1b53775ee0f" (UID: "5338b461-6b7c-474b-8458-d1b53775ee0f"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:09.177400 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:09.177334    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5338b461-6b7c-474b-8458-d1b53775ee0f-trusted-ca-bundle" pod "5338b461-6b7c-474b-8458-d1b53775ee0f" (UID: "5338b461-6b7c-474b-8458-d1b53775ee0f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:09.177452 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:09.177412    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5338b461-6b7c-474b-8458-d1b53775ee0f-service-ca-bundle" pod "5338b461-6b7c-474b-8458-d1b53775ee0f" (UID: "5338b461-6b7c-474b-8458-d1b53775ee0f"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:09.179621 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:09.179598    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5338b461-6b7c-474b-8458-d1b53775ee0f-kube-api-access-4xrrr" pod "5338b461-6b7c-474b-8458-d1b53775ee0f" (UID: "5338b461-6b7c-474b-8458-d1b53775ee0f"). InnerVolumeSpecName "kube-api-access-4xrrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:09.278042 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:09.278008    5043 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5338b461-6b7c-474b-8458-d1b53775ee0f-archives-path\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:09.278042 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:09.278036    5043 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5338b461-6b7c-474b-8458-d1b53775ee0f-trusted-ca-bundle\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:09.278042 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:09.278046    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xrrr\" (UniqueName: \"kubernetes.io/projected/5338b461-6b7c-474b-8458-d1b53775ee0f-kube-api-access-4xrrr\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:09.278263 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:09.278056    5043 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5338b461-6b7c-474b-8458-d1b53775ee0f-service-ca-bundle\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:10.060683 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:10.060641    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-2lcwk-gjv8f" event={"ID":"5338b461-6b7c-474b-8458-d1b53775ee0f","Type":"ContainerDied","Data":"487fa79e6ceaed2223a66678071091aae7f373b853288eb4e2de23616b76a96b"}
Oct 30 19:17:10.060683 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:10.060680    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-2lcwk-gjv8f"
Oct 30 19:17:10.061250 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:10.060778    5043 scope.go:122] "RemoveContainer" containerID="6081dbd078e020eb140d07eec8006a13c5f61e5409c51c7e49470b00cd9133f6"
Oct 30 19:17:10.085926 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:10.085900    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-2lcwk-gjv8f"]
Oct 30 19:17:10.092799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:10.092778    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-2lcwk-gjv8f"]
Oct 30 19:17:10.643197 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:10.643162    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5338b461-6b7c-474b-8458-d1b53775ee0f" path="/var/lib/kubelet/pods/5338b461-6b7c-474b-8458-d1b53775ee0f/volumes"
Oct 30 19:17:24.019570 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:24.019505    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-748f8689f4-6lwjn" podUID="9f302411-01be-438c-a30c-70c944afd630" containerName="registry" containerID="cri-o://8d18bc87003f8e30d5f4e70e8876ce3b391fea1dd1c19e63fbdaa5571766630b" gracePeriod=30
Oct 30 19:17:24.299828 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:24.299730    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:17:24.405871 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:24.405826    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f302411-01be-438c-a30c-70c944afd630-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9f302411-01be-438c-a30c-70c944afd630-ca-trust-extracted\") pod \"9f302411-01be-438c-a30c-70c944afd630\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") "
Oct 30 19:17:24.405871 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:24.405875    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f302411-01be-438c-a30c-70c944afd630-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9f302411-01be-438c-a30c-70c944afd630-trusted-ca\") pod \"9f302411-01be-438c-a30c-70c944afd630\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") "
Oct 30 19:17:24.406076 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:24.405915    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f302411-01be-438c-a30c-70c944afd630-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9f302411-01be-438c-a30c-70c944afd630-registry-certificates\") pod \"9f302411-01be-438c-a30c-70c944afd630\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") "
Oct 30 19:17:24.407091 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:24.406364    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f302411-01be-438c-a30c-70c944afd630-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9f302411-01be-438c-a30c-70c944afd630-image-registry-private-configuration\") pod \"9f302411-01be-438c-a30c-70c944afd630\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") "
Oct 30 19:17:24.407091 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:24.406470    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-kube-api-access-hpb5t\" (UniqueName: \"kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-kube-api-access-hpb5t\") pod \"9f302411-01be-438c-a30c-70c944afd630\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") "
Oct 30 19:17:24.407091 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:24.406512    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f302411-01be-438c-a30c-70c944afd630-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9f302411-01be-438c-a30c-70c944afd630-installation-pull-secrets\") pod \"9f302411-01be-438c-a30c-70c944afd630\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") "
Oct 30 19:17:24.407091 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:24.406582    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-registry-tls\" (UniqueName: \"kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-registry-tls\") pod \"9f302411-01be-438c-a30c-70c944afd630\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") "
Oct 30 19:17:24.407091 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:24.406688    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-bound-sa-token\") pod \"9f302411-01be-438c-a30c-70c944afd630\" (UID: \"9f302411-01be-438c-a30c-70c944afd630\") "
Oct 30 19:17:24.407091 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:24.406711    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f302411-01be-438c-a30c-70c944afd630-registry-certificates" pod "9f302411-01be-438c-a30c-70c944afd630" (UID: "9f302411-01be-438c-a30c-70c944afd630"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:24.407091 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:24.406719    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f302411-01be-438c-a30c-70c944afd630-trusted-ca" pod "9f302411-01be-438c-a30c-70c944afd630" (UID: "9f302411-01be-438c-a30c-70c944afd630"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:24.409874 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:24.408088    5043 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9f302411-01be-438c-a30c-70c944afd630-trusted-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:24.409874 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:24.408117    5043 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9f302411-01be-438c-a30c-70c944afd630-registry-certificates\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:24.414758 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:24.414637    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f302411-01be-438c-a30c-70c944afd630-image-registry-private-configuration" pod "9f302411-01be-438c-a30c-70c944afd630" (UID: "9f302411-01be-438c-a30c-70c944afd630"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:24.414959 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:24.414917    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f302411-01be-438c-a30c-70c944afd630-installation-pull-secrets" pod "9f302411-01be-438c-a30c-70c944afd630" (UID: "9f302411-01be-438c-a30c-70c944afd630"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:24.415070 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:24.414978    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-bound-sa-token" pod "9f302411-01be-438c-a30c-70c944afd630" (UID: "9f302411-01be-438c-a30c-70c944afd630"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:24.415070 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:24.415027    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-registry-tls" pod "9f302411-01be-438c-a30c-70c944afd630" (UID: "9f302411-01be-438c-a30c-70c944afd630"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:24.415070 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:24.415052    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-kube-api-access-hpb5t" pod "9f302411-01be-438c-a30c-70c944afd630" (UID: "9f302411-01be-438c-a30c-70c944afd630"). InnerVolumeSpecName "kube-api-access-hpb5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:24.416819 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:24.416798    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f302411-01be-438c-a30c-70c944afd630-ca-trust-extracted" pod "9f302411-01be-438c-a30c-70c944afd630" (UID: "9f302411-01be-438c-a30c-70c944afd630"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:24.509051 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:24.509020    5043 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9f302411-01be-438c-a30c-70c944afd630-image-registry-private-configuration\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:24.509051 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:24.509046    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpb5t\" (UniqueName: \"kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-kube-api-access-hpb5t\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:24.509051 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:24.509060    5043 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9f302411-01be-438c-a30c-70c944afd630-installation-pull-secrets\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:24.509278 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:24.509071    5043 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-registry-tls\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:24.509278 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:24.509080    5043 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f302411-01be-438c-a30c-70c944afd630-bound-sa-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:24.509278 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:24.509089    5043 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9f302411-01be-438c-a30c-70c944afd630-ca-trust-extracted\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:24.976034 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:17:24.975998    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:17:25.125335 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:25.125305    5043 generic.go:356] "Generic (PLEG): container finished" podID="353ebfb4-7e11-41de-b992-277443f56960" containerID="4b81bc0bd6efd1290bc9ea02d3572f3b8ca96bb1198b289a762c5933e0449b4f" exitCode=0
Oct 30 19:17:25.125826 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:25.125385    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-62bhw" event={"ID":"353ebfb4-7e11-41de-b992-277443f56960","Type":"ContainerDied","Data":"4b81bc0bd6efd1290bc9ea02d3572f3b8ca96bb1198b289a762c5933e0449b4f"}
Oct 30 19:17:25.125826 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:25.125770    5043 scope.go:122] "RemoveContainer" containerID="4b81bc0bd6efd1290bc9ea02d3572f3b8ca96bb1198b289a762c5933e0449b4f"
Oct 30 19:17:25.126663 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:25.126646    5043 generic.go:356] "Generic (PLEG): container finished" podID="9f302411-01be-438c-a30c-70c944afd630" containerID="8d18bc87003f8e30d5f4e70e8876ce3b391fea1dd1c19e63fbdaa5571766630b" exitCode=0
Oct 30 19:17:25.126773 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:25.126726    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-748f8689f4-6lwjn" event={"ID":"9f302411-01be-438c-a30c-70c944afd630","Type":"ContainerDied","Data":"8d18bc87003f8e30d5f4e70e8876ce3b391fea1dd1c19e63fbdaa5571766630b"}
Oct 30 19:17:25.126882 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:25.126793    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-748f8689f4-6lwjn" event={"ID":"9f302411-01be-438c-a30c-70c944afd630","Type":"ContainerDied","Data":"247bbb32dc0ab7d347f4a2bba54493ab310f9e9a0a5529143bbc1bd929ef6a77"}
Oct 30 19:17:25.126882 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:25.126820    5043 scope.go:122] "RemoveContainer" containerID="8d18bc87003f8e30d5f4e70e8876ce3b391fea1dd1c19e63fbdaa5571766630b"
Oct 30 19:17:25.126882 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:25.126843    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-748f8689f4-6lwjn"
Oct 30 19:17:25.135152 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:25.135133    5043 scope.go:122] "RemoveContainer" containerID="8d18bc87003f8e30d5f4e70e8876ce3b391fea1dd1c19e63fbdaa5571766630b"
Oct 30 19:17:25.135419 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:17:25.135401    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d18bc87003f8e30d5f4e70e8876ce3b391fea1dd1c19e63fbdaa5571766630b\": container with ID starting with 8d18bc87003f8e30d5f4e70e8876ce3b391fea1dd1c19e63fbdaa5571766630b not found: ID does not exist" containerID="8d18bc87003f8e30d5f4e70e8876ce3b391fea1dd1c19e63fbdaa5571766630b"
Oct 30 19:17:25.135460 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:25.135428    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d18bc87003f8e30d5f4e70e8876ce3b391fea1dd1c19e63fbdaa5571766630b"} err="failed to get container status \"8d18bc87003f8e30d5f4e70e8876ce3b391fea1dd1c19e63fbdaa5571766630b\": rpc error: code = NotFound desc = could not find container \"8d18bc87003f8e30d5f4e70e8876ce3b391fea1dd1c19e63fbdaa5571766630b\": container with ID starting with 8d18bc87003f8e30d5f4e70e8876ce3b391fea1dd1c19e63fbdaa5571766630b not found: ID does not exist"
Oct 30 19:17:25.160350 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:25.160318    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-748f8689f4-6lwjn"]
Oct 30 19:17:25.164592 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:25.164562    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-748f8689f4-6lwjn"]
Oct 30 19:17:26.132873 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:26.132831    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-62bhw" event={"ID":"353ebfb4-7e11-41de-b992-277443f56960","Type":"ContainerStarted","Data":"66435329fd5959c4282340c98af1246414ebbcdab296082eb1ed69c62b0098d1"}
Oct 30 19:17:26.642755 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:17:26.642714    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f302411-01be-438c-a30c-70c944afd630" path="/var/lib/kubelet/pods/9f302411-01be-438c-a30c-70c944afd630/volumes"
Oct 30 19:17:54.975233 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:17:54.975195    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:18:02.284934 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:18:02.284906    5043 generic.go:356] "Generic (PLEG): container finished" podID="030d88d7-44c3-4945-923e-1e57c2ffd18e" containerID="2318bcf1dcdff54f185aa4d3631ad9e599556918079a229669345c5111797f8c" exitCode=0
Oct 30 19:18:02.285251 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:18:02.284978    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-tqp8q-5cb4z" event={"ID":"030d88d7-44c3-4945-923e-1e57c2ffd18e","Type":"ContainerDied","Data":"2318bcf1dcdff54f185aa4d3631ad9e599556918079a229669345c5111797f8c"}
Oct 30 19:18:03.424528 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:18:03.424505    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-tqp8q-5cb4z"
Oct 30 19:18:03.437410 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:18:03.437390    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/030d88d7-44c3-4945-923e-1e57c2ffd18e-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/030d88d7-44c3-4945-923e-1e57c2ffd18e-archives-path\") pod \"030d88d7-44c3-4945-923e-1e57c2ffd18e\" (UID: \"030d88d7-44c3-4945-923e-1e57c2ffd18e\") "
Oct 30 19:18:03.437530 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:18:03.437422    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/030d88d7-44c3-4945-923e-1e57c2ffd18e-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/030d88d7-44c3-4945-923e-1e57c2ffd18e-trusted-ca-bundle\") pod \"030d88d7-44c3-4945-923e-1e57c2ffd18e\" (UID: \"030d88d7-44c3-4945-923e-1e57c2ffd18e\") "
Oct 30 19:18:03.437530 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:18:03.437449    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/030d88d7-44c3-4945-923e-1e57c2ffd18e-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/030d88d7-44c3-4945-923e-1e57c2ffd18e-service-ca-bundle\") pod \"030d88d7-44c3-4945-923e-1e57c2ffd18e\" (UID: \"030d88d7-44c3-4945-923e-1e57c2ffd18e\") "
Oct 30 19:18:03.437530 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:18:03.437484    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/030d88d7-44c3-4945-923e-1e57c2ffd18e-kube-api-access-llx9b\" (UniqueName: \"kubernetes.io/projected/030d88d7-44c3-4945-923e-1e57c2ffd18e-kube-api-access-llx9b\") pod \"030d88d7-44c3-4945-923e-1e57c2ffd18e\" (UID: \"030d88d7-44c3-4945-923e-1e57c2ffd18e\") "
Oct 30 19:18:03.437792 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:18:03.437695    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/030d88d7-44c3-4945-923e-1e57c2ffd18e-archives-path" pod "030d88d7-44c3-4945-923e-1e57c2ffd18e" (UID: "030d88d7-44c3-4945-923e-1e57c2ffd18e"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:03.437870 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:18:03.437852    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/030d88d7-44c3-4945-923e-1e57c2ffd18e-trusted-ca-bundle" pod "030d88d7-44c3-4945-923e-1e57c2ffd18e" (UID: "030d88d7-44c3-4945-923e-1e57c2ffd18e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:03.437926 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:18:03.437856    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/030d88d7-44c3-4945-923e-1e57c2ffd18e-service-ca-bundle" pod "030d88d7-44c3-4945-923e-1e57c2ffd18e" (UID: "030d88d7-44c3-4945-923e-1e57c2ffd18e"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:03.439685 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:18:03.439665    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/030d88d7-44c3-4945-923e-1e57c2ffd18e-kube-api-access-llx9b" pod "030d88d7-44c3-4945-923e-1e57c2ffd18e" (UID: "030d88d7-44c3-4945-923e-1e57c2ffd18e"). InnerVolumeSpecName "kube-api-access-llx9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:03.538806 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:18:03.538772    5043 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/030d88d7-44c3-4945-923e-1e57c2ffd18e-archives-path\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:03.538806 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:18:03.538799    5043 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/030d88d7-44c3-4945-923e-1e57c2ffd18e-trusted-ca-bundle\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:03.538806 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:18:03.538810    5043 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/030d88d7-44c3-4945-923e-1e57c2ffd18e-service-ca-bundle\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:03.539037 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:18:03.538819    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llx9b\" (UniqueName: \"kubernetes.io/projected/030d88d7-44c3-4945-923e-1e57c2ffd18e-kube-api-access-llx9b\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:04.294132 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:18:04.294090    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-tqp8q-5cb4z" event={"ID":"030d88d7-44c3-4945-923e-1e57c2ffd18e","Type":"ContainerDied","Data":"4128a3e88594cb1b3b1a35fee8fd68c9a0bc0625b8774db6006d00222181c6f2"}
Oct 30 19:18:04.294306 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:18:04.294193    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4128a3e88594cb1b3b1a35fee8fd68c9a0bc0625b8774db6006d00222181c6f2"
Oct 30 19:18:04.294306 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:18:04.294113    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-tqp8q-5cb4z"
Oct 30 19:18:24.975191 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:18:24.975154    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:18:54.975179 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:18:54.975147    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:19:24.975124 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:19:24.975084    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:19:54.975566 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:19:54.975529    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:19:58.588285 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.588247    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h"]
Oct 30 19:19:58.588646 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.588577    5043 cpu_manager.go:414] "containerMap: removing container" podUID="5338b461-6b7c-474b-8458-d1b53775ee0f" containerName="insights-gathering"
Oct 30 19:19:58.588646 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.588588    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5338b461-6b7c-474b-8458-d1b53775ee0f" containerName="insights-gathering"
Oct 30 19:19:58.588646 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.588606    5043 cpu_manager.go:414] "containerMap: removing container" podUID="9f302411-01be-438c-a30c-70c944afd630" containerName="registry"
Oct 30 19:19:58.588646 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.588611    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f302411-01be-438c-a30c-70c944afd630" containerName="registry"
Oct 30 19:19:58.588646 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.588625    5043 cpu_manager.go:414] "containerMap: removing container" podUID="a64211d0-1c48-41d2-b254-4ae84d5097d7" containerName="registry"
Oct 30 19:19:58.588646 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.588630    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a64211d0-1c48-41d2-b254-4ae84d5097d7" containerName="registry"
Oct 30 19:19:58.588646 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.588635    5043 cpu_manager.go:414] "containerMap: removing container" podUID="030d88d7-44c3-4945-923e-1e57c2ffd18e" containerName="insights-gathering"
Oct 30 19:19:58.588646 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.588640    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="030d88d7-44c3-4945-923e-1e57c2ffd18e" containerName="insights-gathering"
Oct 30 19:19:58.588906 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.588725    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f302411-01be-438c-a30c-70c944afd630" containerName="registry"
Oct 30 19:19:58.588906 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.588755    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="a64211d0-1c48-41d2-b254-4ae84d5097d7" containerName="registry"
Oct 30 19:19:58.588906 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.588761    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="030d88d7-44c3-4945-923e-1e57c2ffd18e" containerName="insights-gathering"
Oct 30 19:19:58.588906 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.588770    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="5338b461-6b7c-474b-8458-d1b53775ee0f" containerName="insights-gathering"
Oct 30 19:19:58.592094 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.592070    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h"
Oct 30 19:19:58.605087 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.605053    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 19:19:58.610245 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.610223    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-dr26k\""
Oct 30 19:19:58.619172 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.619146    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h"]
Oct 30 19:19:58.754434 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.754402    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0feb376c-3dc2-4831-a348-abaa09adade6-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h\" (UID: \"0feb376c-3dc2-4831-a348-abaa09adade6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h"
Oct 30 19:19:58.754434 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.754439    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0feb376c-3dc2-4831-a348-abaa09adade6-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h\" (UID: \"0feb376c-3dc2-4831-a348-abaa09adade6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h"
Oct 30 19:19:58.754669 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.754459    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzfwm\" (UniqueName: \"kubernetes.io/projected/0feb376c-3dc2-4831-a348-abaa09adade6-kube-api-access-jzfwm\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h\" (UID: \"0feb376c-3dc2-4831-a348-abaa09adade6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h"
Oct 30 19:19:58.754669 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.754521    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0feb376c-3dc2-4831-a348-abaa09adade6-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h\" (UID: \"0feb376c-3dc2-4831-a348-abaa09adade6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h"
Oct 30 19:19:58.754669 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.754573    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0feb376c-3dc2-4831-a348-abaa09adade6-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h\" (UID: \"0feb376c-3dc2-4831-a348-abaa09adade6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h"
Oct 30 19:19:58.754669 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.754637    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0feb376c-3dc2-4831-a348-abaa09adade6-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h\" (UID: \"0feb376c-3dc2-4831-a348-abaa09adade6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h"
Oct 30 19:19:58.754842 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.754672    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0feb376c-3dc2-4831-a348-abaa09adade6-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h\" (UID: \"0feb376c-3dc2-4831-a348-abaa09adade6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h"
Oct 30 19:19:58.754842 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.754824    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0feb376c-3dc2-4831-a348-abaa09adade6-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h\" (UID: \"0feb376c-3dc2-4831-a348-abaa09adade6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h"
Oct 30 19:19:58.754901 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.754866    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0feb376c-3dc2-4831-a348-abaa09adade6-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h\" (UID: \"0feb376c-3dc2-4831-a348-abaa09adade6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h"
Oct 30 19:19:58.855926 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.855841    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0feb376c-3dc2-4831-a348-abaa09adade6-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h\" (UID: \"0feb376c-3dc2-4831-a348-abaa09adade6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h"
Oct 30 19:19:58.855926 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.855892    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0feb376c-3dc2-4831-a348-abaa09adade6-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h\" (UID: \"0feb376c-3dc2-4831-a348-abaa09adade6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h"
Oct 30 19:19:58.855926 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.855918    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0feb376c-3dc2-4831-a348-abaa09adade6-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h\" (UID: \"0feb376c-3dc2-4831-a348-abaa09adade6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h"
Oct 30 19:19:58.856173 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.855947    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzfwm\" (UniqueName: \"kubernetes.io/projected/0feb376c-3dc2-4831-a348-abaa09adade6-kube-api-access-jzfwm\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h\" (UID: \"0feb376c-3dc2-4831-a348-abaa09adade6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h"
Oct 30 19:19:58.856173 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.855976    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0feb376c-3dc2-4831-a348-abaa09adade6-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h\" (UID: \"0feb376c-3dc2-4831-a348-abaa09adade6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h"
Oct 30 19:19:58.856173 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.856005    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0feb376c-3dc2-4831-a348-abaa09adade6-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h\" (UID: \"0feb376c-3dc2-4831-a348-abaa09adade6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h"
Oct 30 19:19:58.856173 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.856031    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0feb376c-3dc2-4831-a348-abaa09adade6-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h\" (UID: \"0feb376c-3dc2-4831-a348-abaa09adade6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h"
Oct 30 19:19:58.856173 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.856059    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0feb376c-3dc2-4831-a348-abaa09adade6-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h\" (UID: \"0feb376c-3dc2-4831-a348-abaa09adade6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h"
Oct 30 19:19:58.856173 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.856119    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0feb376c-3dc2-4831-a348-abaa09adade6-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h\" (UID: \"0feb376c-3dc2-4831-a348-abaa09adade6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h"
Oct 30 19:19:58.856478 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.856335    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0feb376c-3dc2-4831-a348-abaa09adade6-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h\" (UID: \"0feb376c-3dc2-4831-a348-abaa09adade6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h"
Oct 30 19:19:58.856478 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.856413    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0feb376c-3dc2-4831-a348-abaa09adade6-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h\" (UID: \"0feb376c-3dc2-4831-a348-abaa09adade6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h"
Oct 30 19:19:58.856652 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.856488    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0feb376c-3dc2-4831-a348-abaa09adade6-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h\" (UID: \"0feb376c-3dc2-4831-a348-abaa09adade6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h"
Oct 30 19:19:58.856652 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.856620    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0feb376c-3dc2-4831-a348-abaa09adade6-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h\" (UID: \"0feb376c-3dc2-4831-a348-abaa09adade6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h"
Oct 30 19:19:58.856918 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.856895    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0feb376c-3dc2-4831-a348-abaa09adade6-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h\" (UID: \"0feb376c-3dc2-4831-a348-abaa09adade6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h"
Oct 30 19:19:58.858623 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.858598    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0feb376c-3dc2-4831-a348-abaa09adade6-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h\" (UID: \"0feb376c-3dc2-4831-a348-abaa09adade6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h"
Oct 30 19:19:58.858754 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.858599    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0feb376c-3dc2-4831-a348-abaa09adade6-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h\" (UID: \"0feb376c-3dc2-4831-a348-abaa09adade6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h"
Oct 30 19:19:58.867531 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.867507    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0feb376c-3dc2-4831-a348-abaa09adade6-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h\" (UID: \"0feb376c-3dc2-4831-a348-abaa09adade6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h"
Oct 30 19:19:58.868017 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.867999    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzfwm\" (UniqueName: \"kubernetes.io/projected/0feb376c-3dc2-4831-a348-abaa09adade6-kube-api-access-jzfwm\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h\" (UID: \"0feb376c-3dc2-4831-a348-abaa09adade6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h"
Oct 30 19:19:58.874274 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.874248    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71r58mz"]
Oct 30 19:19:58.883711 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.883684    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71r58mz"
Oct 30 19:19:58.887573 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.887548    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71r58mz"]
Oct 30 19:19:58.887977 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.887954    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 19:19:58.887977 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.887971    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 19:19:58.888342 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.888323    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-wcgjd\""
Oct 30 19:19:58.911115 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:58.911086    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h"
Oct 30 19:19:59.058516 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:59.058477    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grvpc\" (UniqueName: \"kubernetes.io/projected/274ec961-719c-4bd3-8b4a-819c1b7b2cfa-kube-api-access-grvpc\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71r58mz\" (UID: \"274ec961-719c-4bd3-8b4a-819c1b7b2cfa\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71r58mz"
Oct 30 19:19:59.058694 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:59.058524    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/274ec961-719c-4bd3-8b4a-819c1b7b2cfa-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71r58mz\" (UID: \"274ec961-719c-4bd3-8b4a-819c1b7b2cfa\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71r58mz"
Oct 30 19:19:59.058694 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:59.058645    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/274ec961-719c-4bd3-8b4a-819c1b7b2cfa-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71r58mz\" (UID: \"274ec961-719c-4bd3-8b4a-819c1b7b2cfa\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71r58mz"
Oct 30 19:19:59.159494 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:59.159405    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/274ec961-719c-4bd3-8b4a-819c1b7b2cfa-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71r58mz\" (UID: \"274ec961-719c-4bd3-8b4a-819c1b7b2cfa\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71r58mz"
Oct 30 19:19:59.159494 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:59.159471    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grvpc\" (UniqueName: \"kubernetes.io/projected/274ec961-719c-4bd3-8b4a-819c1b7b2cfa-kube-api-access-grvpc\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71r58mz\" (UID: \"274ec961-719c-4bd3-8b4a-819c1b7b2cfa\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71r58mz"
Oct 30 19:19:59.159686 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:59.159497    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/274ec961-719c-4bd3-8b4a-819c1b7b2cfa-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71r58mz\" (UID: \"274ec961-719c-4bd3-8b4a-819c1b7b2cfa\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71r58mz"
Oct 30 19:19:59.159848 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:59.159830    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/274ec961-719c-4bd3-8b4a-819c1b7b2cfa-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71r58mz\" (UID: \"274ec961-719c-4bd3-8b4a-819c1b7b2cfa\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71r58mz"
Oct 30 19:19:59.159895 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:59.159866    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/274ec961-719c-4bd3-8b4a-819c1b7b2cfa-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71r58mz\" (UID: \"274ec961-719c-4bd3-8b4a-819c1b7b2cfa\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71r58mz"
Oct 30 19:19:59.170451 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:59.170420    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grvpc\" (UniqueName: \"kubernetes.io/projected/274ec961-719c-4bd3-8b4a-819c1b7b2cfa-kube-api-access-grvpc\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71r58mz\" (UID: \"274ec961-719c-4bd3-8b4a-819c1b7b2cfa\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71r58mz"
Oct 30 19:19:59.187172 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:59.187136    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h"]
Oct 30 19:19:59.190088 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:19:59.190060    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0feb376c_3dc2_4831_a348_abaa09adade6.slice/crio-41a259fe28033df891037d1d31c770e4424bc6a6db2cc46a5af30f68b1dfe526 WatchSource:0}: Error finding container 41a259fe28033df891037d1d31c770e4424bc6a6db2cc46a5af30f68b1dfe526: Status 404 returned error can't find the container with id 41a259fe28033df891037d1d31c770e4424bc6a6db2cc46a5af30f68b1dfe526
Oct 30 19:19:59.202540 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:59.202520    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71r58mz"
Oct 30 19:19:59.476285 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:59.476257    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71r58mz"]
Oct 30 19:19:59.479236 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:19:59.479209    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod274ec961_719c_4bd3_8b4a_819c1b7b2cfa.slice/crio-913e4ed79ec11bf4f5b21a15f2067bb9df159579a632ffa14711266caca79ca2 WatchSource:0}: Error finding container 913e4ed79ec11bf4f5b21a15f2067bb9df159579a632ffa14711266caca79ca2: Status 404 returned error can't find the container with id 913e4ed79ec11bf4f5b21a15f2067bb9df159579a632ffa14711266caca79ca2
Oct 30 19:19:59.757322 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:59.757227    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71r58mz" event={"ID":"274ec961-719c-4bd3-8b4a-819c1b7b2cfa","Type":"ContainerStarted","Data":"913e4ed79ec11bf4f5b21a15f2067bb9df159579a632ffa14711266caca79ca2"}
Oct 30 19:19:59.758439 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:19:59.758416    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h" event={"ID":"0feb376c-3dc2-4831-a348-abaa09adade6","Type":"ContainerStarted","Data":"41a259fe28033df891037d1d31c770e4424bc6a6db2cc46a5af30f68b1dfe526"}
Oct 30 19:20:02.460976 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:02.460940    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:20:02.461421 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:02.461007    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:20:02.461421 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:02.461037    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:20:02.775657 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:02.775618    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h" event={"ID":"0feb376c-3dc2-4831-a348-abaa09adade6","Type":"ContainerStarted","Data":"e76ada690a33d7721790c438e80a08c40b917685dde2ea1e3b946e1d8697ffa9"}
Oct 30 19:20:02.800717 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:02.800651    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h" podStartSLOduration=1.532551632 podStartE2EDuration="4.800630738s" totalImagesPullingTime="3.268079106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:02.798312759 +0000 UTC m=+222.309372724" watchObservedRunningTime="2026-10-30 19:20:02.800630738 +0000 UTC m=+222.311690707"
Oct 30 19:20:02.911876 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:02.911839    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h"
Oct 30 19:20:03.917243 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:03.917214    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h"
Oct 30 19:20:04.787076 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:04.787036    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h"
Oct 30 19:20:04.788232 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:04.788209    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h"
Oct 30 19:20:05.792612 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:05.792578    5043 generic.go:356] "Generic (PLEG): container finished" podID="274ec961-719c-4bd3-8b4a-819c1b7b2cfa" containerID="4a6aed7059f667fdf63906bf7365dec13eedbfa526da5e410e688366706df8e1" exitCode=0
Oct 30 19:20:05.793082 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:05.792658    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71r58mz" event={"ID":"274ec961-719c-4bd3-8b4a-819c1b7b2cfa","Type":"ContainerDied","Data":"4a6aed7059f667fdf63906bf7365dec13eedbfa526da5e410e688366706df8e1"}
Oct 30 19:20:06.799208 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:06.799174    5043 generic.go:356] "Generic (PLEG): container finished" podID="274ec961-719c-4bd3-8b4a-819c1b7b2cfa" containerID="246cf36ca35a420cec1c498892cacf3c97c8eb931be8adf3ce05350a67ef5d61" exitCode=0
Oct 30 19:20:06.799593 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:06.799265    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71r58mz" event={"ID":"274ec961-719c-4bd3-8b4a-819c1b7b2cfa","Type":"ContainerDied","Data":"246cf36ca35a420cec1c498892cacf3c97c8eb931be8adf3ce05350a67ef5d61"}
Oct 30 19:20:14.838302 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:14.838262    5043 generic.go:356] "Generic (PLEG): container finished" podID="274ec961-719c-4bd3-8b4a-819c1b7b2cfa" containerID="5f5b79c8840c72fc9f01105765bc2d4376653a4d7c8124f4dcb62318e48bfde8" exitCode=0
Oct 30 19:20:14.838672 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:14.838338    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71r58mz" event={"ID":"274ec961-719c-4bd3-8b4a-819c1b7b2cfa","Type":"ContainerDied","Data":"5f5b79c8840c72fc9f01105765bc2d4376653a4d7c8124f4dcb62318e48bfde8"}
Oct 30 19:20:15.972749 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:15.972708    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71r58mz"
Oct 30 19:20:16.013301 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:16.013268    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/274ec961-719c-4bd3-8b4a-819c1b7b2cfa-bundle\" (UniqueName: \"kubernetes.io/empty-dir/274ec961-719c-4bd3-8b4a-819c1b7b2cfa-bundle\") pod \"274ec961-719c-4bd3-8b4a-819c1b7b2cfa\" (UID: \"274ec961-719c-4bd3-8b4a-819c1b7b2cfa\") "
Oct 30 19:20:16.013478 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:16.013314    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/274ec961-719c-4bd3-8b4a-819c1b7b2cfa-util\" (UniqueName: \"kubernetes.io/empty-dir/274ec961-719c-4bd3-8b4a-819c1b7b2cfa-util\") pod \"274ec961-719c-4bd3-8b4a-819c1b7b2cfa\" (UID: \"274ec961-719c-4bd3-8b4a-819c1b7b2cfa\") "
Oct 30 19:20:16.013478 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:16.013358    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/274ec961-719c-4bd3-8b4a-819c1b7b2cfa-kube-api-access-grvpc\" (UniqueName: \"kubernetes.io/projected/274ec961-719c-4bd3-8b4a-819c1b7b2cfa-kube-api-access-grvpc\") pod \"274ec961-719c-4bd3-8b4a-819c1b7b2cfa\" (UID: \"274ec961-719c-4bd3-8b4a-819c1b7b2cfa\") "
Oct 30 19:20:16.013930 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:16.013901    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/274ec961-719c-4bd3-8b4a-819c1b7b2cfa-bundle" pod "274ec961-719c-4bd3-8b4a-819c1b7b2cfa" (UID: "274ec961-719c-4bd3-8b4a-819c1b7b2cfa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:16.015751 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:16.015708    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/274ec961-719c-4bd3-8b4a-819c1b7b2cfa-kube-api-access-grvpc" pod "274ec961-719c-4bd3-8b4a-819c1b7b2cfa" (UID: "274ec961-719c-4bd3-8b4a-819c1b7b2cfa"). InnerVolumeSpecName "kube-api-access-grvpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:16.017763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:16.017721    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/274ec961-719c-4bd3-8b4a-819c1b7b2cfa-util" pod "274ec961-719c-4bd3-8b4a-819c1b7b2cfa" (UID: "274ec961-719c-4bd3-8b4a-819c1b7b2cfa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:16.114549 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:16.114476    5043 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/274ec961-719c-4bd3-8b4a-819c1b7b2cfa-util\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:16.114549 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:16.114513    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grvpc\" (UniqueName: \"kubernetes.io/projected/274ec961-719c-4bd3-8b4a-819c1b7b2cfa-kube-api-access-grvpc\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:16.114549 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:16.114523    5043 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/274ec961-719c-4bd3-8b4a-819c1b7b2cfa-bundle\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:16.848697 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:16.848658    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71r58mz" event={"ID":"274ec961-719c-4bd3-8b4a-819c1b7b2cfa","Type":"ContainerDied","Data":"913e4ed79ec11bf4f5b21a15f2067bb9df159579a632ffa14711266caca79ca2"}
Oct 30 19:20:16.848697 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:16.848704    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="913e4ed79ec11bf4f5b21a15f2067bb9df159579a632ffa14711266caca79ca2"
Oct 30 19:20:16.848968 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:16.848706    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71r58mz"
Oct 30 19:20:21.695358 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:21.695324    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-svlrj"]
Oct 30 19:20:21.695811 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:21.695643    5043 cpu_manager.go:414] "containerMap: removing container" podUID="274ec961-719c-4bd3-8b4a-819c1b7b2cfa" containerName="extract"
Oct 30 19:20:21.695811 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:21.695654    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="274ec961-719c-4bd3-8b4a-819c1b7b2cfa" containerName="extract"
Oct 30 19:20:21.695811 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:21.695675    5043 cpu_manager.go:414] "containerMap: removing container" podUID="274ec961-719c-4bd3-8b4a-819c1b7b2cfa" containerName="util"
Oct 30 19:20:21.695811 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:21.695681    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="274ec961-719c-4bd3-8b4a-819c1b7b2cfa" containerName="util"
Oct 30 19:20:21.695811 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:21.695689    5043 cpu_manager.go:414] "containerMap: removing container" podUID="274ec961-719c-4bd3-8b4a-819c1b7b2cfa" containerName="pull"
Oct 30 19:20:21.695811 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:21.695695    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="274ec961-719c-4bd3-8b4a-819c1b7b2cfa" containerName="pull"
Oct 30 19:20:21.695811 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:21.695766    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="274ec961-719c-4bd3-8b4a-819c1b7b2cfa" containerName="extract"
Oct 30 19:20:21.747266 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:21.747236    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-svlrj"
Oct 30 19:20:21.760284 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:21.760254    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 19:20:21.760615 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:21.760598    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 19:20:21.760782 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:21.760762    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-hbls9\""
Oct 30 19:20:21.760900 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:21.760789    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 19:20:21.760941 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:21.760905    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 19:20:21.762924 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:21.762905    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 19:20:21.779953 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:21.779928    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-svlrj"]
Oct 30 19:20:21.862814 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:21.862782    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3d93bc51-b50b-4595-b6c5-7494bfaec7a9-webhook-cert\") pod \"opendatahub-operator-controller-manager-68fb5c5984-svlrj\" (UID: \"3d93bc51-b50b-4595-b6c5-7494bfaec7a9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-svlrj"
Oct 30 19:20:21.862814 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:21.862826    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z4zc\" (UniqueName: \"kubernetes.io/projected/3d93bc51-b50b-4595-b6c5-7494bfaec7a9-kube-api-access-9z4zc\") pod \"opendatahub-operator-controller-manager-68fb5c5984-svlrj\" (UID: \"3d93bc51-b50b-4595-b6c5-7494bfaec7a9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-svlrj"
Oct 30 19:20:21.863060 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:21.862861    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3d93bc51-b50b-4595-b6c5-7494bfaec7a9-metrics-cert\") pod \"opendatahub-operator-controller-manager-68fb5c5984-svlrj\" (UID: \"3d93bc51-b50b-4595-b6c5-7494bfaec7a9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-svlrj"
Oct 30 19:20:21.863060 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:21.862916    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d93bc51-b50b-4595-b6c5-7494bfaec7a9-tmp\") pod \"opendatahub-operator-controller-manager-68fb5c5984-svlrj\" (UID: \"3d93bc51-b50b-4595-b6c5-7494bfaec7a9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-svlrj"
Oct 30 19:20:21.863060 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:21.863034    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3d93bc51-b50b-4595-b6c5-7494bfaec7a9-manifests\") pod \"opendatahub-operator-controller-manager-68fb5c5984-svlrj\" (UID: \"3d93bc51-b50b-4595-b6c5-7494bfaec7a9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-svlrj"
Oct 30 19:20:21.863171 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:21.863077    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3d93bc51-b50b-4595-b6c5-7494bfaec7a9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68fb5c5984-svlrj\" (UID: \"3d93bc51-b50b-4595-b6c5-7494bfaec7a9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-svlrj"
Oct 30 19:20:21.963453 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:21.963420    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d93bc51-b50b-4595-b6c5-7494bfaec7a9-tmp\") pod \"opendatahub-operator-controller-manager-68fb5c5984-svlrj\" (UID: \"3d93bc51-b50b-4595-b6c5-7494bfaec7a9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-svlrj"
Oct 30 19:20:21.963625 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:21.963484    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3d93bc51-b50b-4595-b6c5-7494bfaec7a9-manifests\") pod \"opendatahub-operator-controller-manager-68fb5c5984-svlrj\" (UID: \"3d93bc51-b50b-4595-b6c5-7494bfaec7a9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-svlrj"
Oct 30 19:20:21.963625 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:21.963547    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3d93bc51-b50b-4595-b6c5-7494bfaec7a9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68fb5c5984-svlrj\" (UID: \"3d93bc51-b50b-4595-b6c5-7494bfaec7a9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-svlrj"
Oct 30 19:20:21.963625 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:21.963575    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3d93bc51-b50b-4595-b6c5-7494bfaec7a9-webhook-cert\") pod \"opendatahub-operator-controller-manager-68fb5c5984-svlrj\" (UID: \"3d93bc51-b50b-4595-b6c5-7494bfaec7a9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-svlrj"
Oct 30 19:20:21.963625 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:21.963597    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z4zc\" (UniqueName: \"kubernetes.io/projected/3d93bc51-b50b-4595-b6c5-7494bfaec7a9-kube-api-access-9z4zc\") pod \"opendatahub-operator-controller-manager-68fb5c5984-svlrj\" (UID: \"3d93bc51-b50b-4595-b6c5-7494bfaec7a9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-svlrj"
Oct 30 19:20:21.963837 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:21.963631    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3d93bc51-b50b-4595-b6c5-7494bfaec7a9-metrics-cert\") pod \"opendatahub-operator-controller-manager-68fb5c5984-svlrj\" (UID: \"3d93bc51-b50b-4595-b6c5-7494bfaec7a9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-svlrj"
Oct 30 19:20:21.964010 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:21.963983    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3d93bc51-b50b-4595-b6c5-7494bfaec7a9-manifests\") pod \"opendatahub-operator-controller-manager-68fb5c5984-svlrj\" (UID: \"3d93bc51-b50b-4595-b6c5-7494bfaec7a9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-svlrj"
Oct 30 19:20:21.966442 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:21.966415    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3d93bc51-b50b-4595-b6c5-7494bfaec7a9-metrics-cert\") pod \"opendatahub-operator-controller-manager-68fb5c5984-svlrj\" (UID: \"3d93bc51-b50b-4595-b6c5-7494bfaec7a9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-svlrj"
Oct 30 19:20:21.966576 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:21.966440    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d93bc51-b50b-4595-b6c5-7494bfaec7a9-tmp\") pod \"opendatahub-operator-controller-manager-68fb5c5984-svlrj\" (UID: \"3d93bc51-b50b-4595-b6c5-7494bfaec7a9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-svlrj"
Oct 30 19:20:21.966576 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:21.966475    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3d93bc51-b50b-4595-b6c5-7494bfaec7a9-webhook-cert\") pod \"opendatahub-operator-controller-manager-68fb5c5984-svlrj\" (UID: \"3d93bc51-b50b-4595-b6c5-7494bfaec7a9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-svlrj"
Oct 30 19:20:21.967017 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:21.966996    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3d93bc51-b50b-4595-b6c5-7494bfaec7a9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68fb5c5984-svlrj\" (UID: \"3d93bc51-b50b-4595-b6c5-7494bfaec7a9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-svlrj"
Oct 30 19:20:21.976137 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:21.976116    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z4zc\" (UniqueName: \"kubernetes.io/projected/3d93bc51-b50b-4595-b6c5-7494bfaec7a9-kube-api-access-9z4zc\") pod \"opendatahub-operator-controller-manager-68fb5c5984-svlrj\" (UID: \"3d93bc51-b50b-4595-b6c5-7494bfaec7a9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-svlrj"
Oct 30 19:20:22.091582 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:22.091545    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-svlrj"
Oct 30 19:20:22.443047 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:22.443021    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-svlrj"]
Oct 30 19:20:22.444814 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:20:22.444788    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d93bc51_b50b_4595_b6c5_7494bfaec7a9.slice/crio-9717c348b839a87392839f3aacebac333f3f3cf8f2bbedb34b4fb3fda5afecfb WatchSource:0}: Error finding container 9717c348b839a87392839f3aacebac333f3f3cf8f2bbedb34b4fb3fda5afecfb: Status 404 returned error can't find the container with id 9717c348b839a87392839f3aacebac333f3f3cf8f2bbedb34b4fb3fda5afecfb
Oct 30 19:20:22.874437 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:22.874400    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-svlrj" event={"ID":"3d93bc51-b50b-4595-b6c5-7494bfaec7a9","Type":"ContainerStarted","Data":"9717c348b839a87392839f3aacebac333f3f3cf8f2bbedb34b4fb3fda5afecfb"}
Oct 30 19:20:24.978868 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:20:24.978833    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:20:26.894356 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:26.894269    5043 generic.go:356] "Generic (PLEG): container finished" podID="3d93bc51-b50b-4595-b6c5-7494bfaec7a9" containerID="825166e0065e9939a81cb45ee7eebd0b16d45d5937e3302448c8beda657fe84e" exitCode=0
Oct 30 19:20:26.894356 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:26.894343    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-svlrj" event={"ID":"3d93bc51-b50b-4595-b6c5-7494bfaec7a9","Type":"ContainerDied","Data":"825166e0065e9939a81cb45ee7eebd0b16d45d5937e3302448c8beda657fe84e"}
Oct 30 19:20:27.900496 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:27.900408    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-svlrj" event={"ID":"3d93bc51-b50b-4595-b6c5-7494bfaec7a9","Type":"ContainerStarted","Data":"cd5206e15841f3ee22999bec75ba768dc9b551dfc052d6ad1a7ee2e623ba6b2c"}
Oct 30 19:20:27.900496 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:27.900485    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-svlrj"
Oct 30 19:20:28.061229 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:28.061178    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-svlrj" podStartSLOduration=2.444534637 podStartE2EDuration="7.061164084s" totalImagesPullingTime="4.213234854s" totalInitContainerRuntime="403.394593ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:28.059363502 +0000 UTC m=+247.570423472" watchObservedRunningTime="2026-10-30 19:20:28.061164084 +0000 UTC m=+247.572224048"
Oct 30 19:20:38.908141 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:38.908109    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-svlrj"
Oct 30 19:20:46.670502 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:46.670469    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-wp2bs"]
Oct 30 19:20:46.673801 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:46.673785    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-wp2bs"
Oct 30 19:20:46.683110 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:46.683091    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-n4fbd\""
Oct 30 19:20:46.683267 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:46.683250    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 19:20:46.683319 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:46.683267    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 19:20:46.692790 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:46.692761    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-wp2bs"]
Oct 30 19:20:46.784016 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:46.783978    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glvgr\" (UniqueName: \"kubernetes.io/projected/4d6a8da0-155a-4d5c-b343-f9efea650d55-kube-api-access-glvgr\") pod \"authorino-operator-7b7bb4bc77-wp2bs\" (UID: \"4d6a8da0-155a-4d5c-b343-f9efea650d55\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-wp2bs"
Oct 30 19:20:46.885311 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:46.885268    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glvgr\" (UniqueName: \"kubernetes.io/projected/4d6a8da0-155a-4d5c-b343-f9efea650d55-kube-api-access-glvgr\") pod \"authorino-operator-7b7bb4bc77-wp2bs\" (UID: \"4d6a8da0-155a-4d5c-b343-f9efea650d55\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-wp2bs"
Oct 30 19:20:46.897478 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:46.897453    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glvgr\" (UniqueName: \"kubernetes.io/projected/4d6a8da0-155a-4d5c-b343-f9efea650d55-kube-api-access-glvgr\") pod \"authorino-operator-7b7bb4bc77-wp2bs\" (UID: \"4d6a8da0-155a-4d5c-b343-f9efea650d55\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-wp2bs"
Oct 30 19:20:46.986644 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:46.986619    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-wp2bs"
Oct 30 19:20:47.267911 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:47.267808    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-wp2bs"]
Oct 30 19:20:47.270922 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:20:47.270891    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d6a8da0_155a_4d5c_b343_f9efea650d55.slice/crio-c3b3539ec6f1e368f24c5460018323511c1bd51bd869efd5de87059ee9078abf WatchSource:0}: Error finding container c3b3539ec6f1e368f24c5460018323511c1bd51bd869efd5de87059ee9078abf: Status 404 returned error can't find the container with id c3b3539ec6f1e368f24c5460018323511c1bd51bd869efd5de87059ee9078abf
Oct 30 19:20:47.983933 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:47.983903    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-wp2bs" event={"ID":"4d6a8da0-155a-4d5c-b343-f9efea650d55","Type":"ContainerStarted","Data":"c3b3539ec6f1e368f24c5460018323511c1bd51bd869efd5de87059ee9078abf"}
Oct 30 19:20:50.999036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:50.999000    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-wp2bs" event={"ID":"4d6a8da0-155a-4d5c-b343-f9efea650d55","Type":"ContainerStarted","Data":"a9da85aab3048570333a26af83a4b3f8de5722a97503e3889c6c300434b4b61f"}
Oct 30 19:20:50.999428 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:50.999106    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-wp2bs"
Oct 30 19:20:51.021864 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:51.021813    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-wp2bs" podStartSLOduration=2.247313866 podStartE2EDuration="5.021799667s" totalImagesPullingTime="2.774485801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:51.020948411 +0000 UTC m=+270.532008375" watchObservedRunningTime="2026-10-30 19:20:51.021799667 +0000 UTC m=+270.532859631"
Oct 30 19:20:51.091276 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:51.091240    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-pbdbn"]
Oct 30 19:20:51.094827 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:51.094807    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pbdbn"
Oct 30 19:20:51.103313 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:51.103294    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-5fbgm\""
Oct 30 19:20:51.105712 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:51.105699    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 30 19:20:51.221925 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:51.221892    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v98q8\" (UniqueName: \"kubernetes.io/projected/3d9b887e-6cc7-4f08-9365-38021d72e48f-kube-api-access-v98q8\") pod \"dns-operator-controller-manager-8557cbdb84-pbdbn\" (UID: \"3d9b887e-6cc7-4f08-9365-38021d72e48f\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pbdbn"
Oct 30 19:20:51.232635 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:51.232604    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-pbdbn"]
Oct 30 19:20:51.322727 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:51.322627    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v98q8\" (UniqueName: \"kubernetes.io/projected/3d9b887e-6cc7-4f08-9365-38021d72e48f-kube-api-access-v98q8\") pod \"dns-operator-controller-manager-8557cbdb84-pbdbn\" (UID: \"3d9b887e-6cc7-4f08-9365-38021d72e48f\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pbdbn"
Oct 30 19:20:51.331862 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:51.331836    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v98q8\" (UniqueName: \"kubernetes.io/projected/3d9b887e-6cc7-4f08-9365-38021d72e48f-kube-api-access-v98q8\") pod \"dns-operator-controller-manager-8557cbdb84-pbdbn\" (UID: \"3d9b887e-6cc7-4f08-9365-38021d72e48f\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pbdbn"
Oct 30 19:20:51.407565 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:51.407529    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pbdbn"
Oct 30 19:20:51.686786 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:51.686758    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-pbdbn"]
Oct 30 19:20:51.690485 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:20:51.690460    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d9b887e_6cc7_4f08_9365_38021d72e48f.slice/crio-61ad680cf6d8f10cb555619ffd3ea77d89a9f2d9033849a4f09e3e1e82fe3a19 WatchSource:0}: Error finding container 61ad680cf6d8f10cb555619ffd3ea77d89a9f2d9033849a4f09e3e1e82fe3a19: Status 404 returned error can't find the container with id 61ad680cf6d8f10cb555619ffd3ea77d89a9f2d9033849a4f09e3e1e82fe3a19
Oct 30 19:20:52.008312 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:52.008278    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pbdbn" event={"ID":"3d9b887e-6cc7-4f08-9365-38021d72e48f","Type":"ContainerStarted","Data":"61ad680cf6d8f10cb555619ffd3ea77d89a9f2d9033849a4f09e3e1e82fe3a19"}
Oct 30 19:20:54.019036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:54.019001    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pbdbn" event={"ID":"3d9b887e-6cc7-4f08-9365-38021d72e48f","Type":"ContainerStarted","Data":"97ad578bc65561a8744a53e639a9afa53ae31afd5fa740205d5e39afe2ab7e26"}
Oct 30 19:20:54.019481 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:54.019148    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pbdbn"
Oct 30 19:20:54.040839 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:54.040777    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pbdbn" podStartSLOduration=0.876021102 podStartE2EDuration="3.040762066s" totalImagesPullingTime="2.164740964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:54.039086753 +0000 UTC m=+273.550146718" watchObservedRunningTime="2026-10-30 19:20:54.040762066 +0000 UTC m=+273.551822022"
Oct 30 19:20:54.771105 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:54.771070    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-9lkpq"]
Oct 30 19:20:54.774909 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:54.774879    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9lkpq"
Oct 30 19:20:54.783775 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:54.783731    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-l5kq7\""
Oct 30 19:20:54.795318 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:54.795297    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-9lkpq"]
Oct 30 19:20:54.954013 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:54.953979    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfrhm\" (UniqueName: \"kubernetes.io/projected/8117eb3d-2697-4f54-bc43-e70935303423-kube-api-access-rfrhm\") pod \"limitador-operator-controller-manager-5778bb4449-9lkpq\" (UID: \"8117eb3d-2697-4f54-bc43-e70935303423\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9lkpq"
Oct 30 19:20:54.975712 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:20:54.975677    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:20:55.055071 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:55.054993    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfrhm\" (UniqueName: \"kubernetes.io/projected/8117eb3d-2697-4f54-bc43-e70935303423-kube-api-access-rfrhm\") pod \"limitador-operator-controller-manager-5778bb4449-9lkpq\" (UID: \"8117eb3d-2697-4f54-bc43-e70935303423\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9lkpq"
Oct 30 19:20:55.064422 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:55.064399    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfrhm\" (UniqueName: \"kubernetes.io/projected/8117eb3d-2697-4f54-bc43-e70935303423-kube-api-access-rfrhm\") pod \"limitador-operator-controller-manager-5778bb4449-9lkpq\" (UID: \"8117eb3d-2697-4f54-bc43-e70935303423\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9lkpq"
Oct 30 19:20:55.087196 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:55.087174    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9lkpq"
Oct 30 19:20:55.409226 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:55.409203    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-9lkpq"]
Oct 30 19:20:55.411348 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:20:55.411319    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8117eb3d_2697_4f54_bc43_e70935303423.slice/crio-f2fe926a15be30e7e7ad08d9b3bb1ca67680b77b476522143d22c6bc3d3d4de2 WatchSource:0}: Error finding container f2fe926a15be30e7e7ad08d9b3bb1ca67680b77b476522143d22c6bc3d3d4de2: Status 404 returned error can't find the container with id f2fe926a15be30e7e7ad08d9b3bb1ca67680b77b476522143d22c6bc3d3d4de2
Oct 30 19:20:56.028642 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:56.028610    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9lkpq" event={"ID":"8117eb3d-2697-4f54-bc43-e70935303423","Type":"ContainerStarted","Data":"f2fe926a15be30e7e7ad08d9b3bb1ca67680b77b476522143d22c6bc3d3d4de2"}
Oct 30 19:20:58.039557 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:58.039517    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9lkpq" event={"ID":"8117eb3d-2697-4f54-bc43-e70935303423","Type":"ContainerStarted","Data":"33c439d38c6598ddcaad4083ef0d9efb50b39442620457b22e199cbe2ef9afb6"}
Oct 30 19:20:58.039945 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:58.039660    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9lkpq"
Oct 30 19:20:58.059300 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:58.059258    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9lkpq" podStartSLOduration=1.617155379 podStartE2EDuration="4.05923127s" totalImagesPullingTime="2.442075891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:58.059097474 +0000 UTC m=+277.570157454" watchObservedRunningTime="2026-10-30 19:20:58.05923127 +0000 UTC m=+277.570291235"
Oct 30 19:20:59.698711 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:59.698677    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sdss4"]
Oct 30 19:20:59.732071 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:59.732041    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sdss4"]
Oct 30 19:20:59.732220 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:59.732144    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sdss4"
Oct 30 19:20:59.740873 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:59.740853    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-p995g\""
Oct 30 19:20:59.796820 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:59.796789    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl78s\" (UniqueName: \"kubernetes.io/projected/3929b4ec-4986-41a3-9fd0-929b3d254cb9-kube-api-access-hl78s\") pod \"kuadrant-operator-controller-manager-678d87766f-sdss4\" (UID: \"3929b4ec-4986-41a3-9fd0-929b3d254cb9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sdss4"
Oct 30 19:20:59.796981 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:59.796852    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3929b4ec-4986-41a3-9fd0-929b3d254cb9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-sdss4\" (UID: \"3929b4ec-4986-41a3-9fd0-929b3d254cb9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sdss4"
Oct 30 19:20:59.897908 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:59.897877    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl78s\" (UniqueName: \"kubernetes.io/projected/3929b4ec-4986-41a3-9fd0-929b3d254cb9-kube-api-access-hl78s\") pod \"kuadrant-operator-controller-manager-678d87766f-sdss4\" (UID: \"3929b4ec-4986-41a3-9fd0-929b3d254cb9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sdss4"
Oct 30 19:20:59.898109 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:59.897921    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3929b4ec-4986-41a3-9fd0-929b3d254cb9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-sdss4\" (UID: \"3929b4ec-4986-41a3-9fd0-929b3d254cb9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sdss4"
Oct 30 19:20:59.898266 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:59.898247    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3929b4ec-4986-41a3-9fd0-929b3d254cb9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-sdss4\" (UID: \"3929b4ec-4986-41a3-9fd0-929b3d254cb9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sdss4"
Oct 30 19:20:59.908875 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:20:59.908852    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl78s\" (UniqueName: \"kubernetes.io/projected/3929b4ec-4986-41a3-9fd0-929b3d254cb9-kube-api-access-hl78s\") pod \"kuadrant-operator-controller-manager-678d87766f-sdss4\" (UID: \"3929b4ec-4986-41a3-9fd0-929b3d254cb9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sdss4"
Oct 30 19:21:00.054615 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:00.054583    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sdss4"
Oct 30 19:21:00.331105 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:00.331019    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sdss4"]
Oct 30 19:21:00.334990 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:21:00.334954    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3929b4ec_4986_41a3_9fd0_929b3d254cb9.slice/crio-08e394b96a67b62fd615af438975f9572a5c707c9a2e5228f020b830c93ccbc8 WatchSource:0}: Error finding container 08e394b96a67b62fd615af438975f9572a5c707c9a2e5228f020b830c93ccbc8: Status 404 returned error can't find the container with id 08e394b96a67b62fd615af438975f9572a5c707c9a2e5228f020b830c93ccbc8
Oct 30 19:21:01.054965 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:01.054930    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sdss4" event={"ID":"3929b4ec-4986-41a3-9fd0-929b3d254cb9","Type":"ContainerStarted","Data":"08e394b96a67b62fd615af438975f9572a5c707c9a2e5228f020b830c93ccbc8"}
Oct 30 19:21:02.011625 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:02.011591    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-wp2bs"
Oct 30 19:21:05.025320 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:05.025292    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pbdbn"
Oct 30 19:21:05.078396 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:05.078364    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sdss4" event={"ID":"3929b4ec-4986-41a3-9fd0-929b3d254cb9","Type":"ContainerStarted","Data":"35a45a169678be98decd092dcd96ff9cf4a5fc83b62e3b15896e52726a524865"}
Oct 30 19:21:05.078564 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:05.078449    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sdss4"
Oct 30 19:21:05.107790 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:05.107718    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sdss4" podStartSLOduration=2.124684501 podStartE2EDuration="6.107703566s" totalImagesPullingTime="3.983019065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:05.106086285 +0000 UTC m=+284.617146250" watchObservedRunningTime="2026-10-30 19:21:05.107703566 +0000 UTC m=+284.618763602"
Oct 30 19:21:06.793721 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:06.793693    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fw65nx"]
Oct 30 19:21:06.797326 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:06.797307    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fw65nx"
Oct 30 19:21:06.809993 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:06.809974    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-smsk9\""
Oct 30 19:21:06.818530 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:06.818508    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fw65nx"]
Oct 30 19:21:06.860144 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:06.860111    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c9fc077-3f23-40d1-ba6e-2575551e01a8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fw65nx\" (UID: \"0c9fc077-3f23-40d1-ba6e-2575551e01a8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fw65nx"
Oct 30 19:21:06.860330 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:06.860152    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c9fc077-3f23-40d1-ba6e-2575551e01a8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fw65nx\" (UID: \"0c9fc077-3f23-40d1-ba6e-2575551e01a8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fw65nx"
Oct 30 19:21:06.860330 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:06.860209    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c9fc077-3f23-40d1-ba6e-2575551e01a8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fw65nx\" (UID: \"0c9fc077-3f23-40d1-ba6e-2575551e01a8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fw65nx"
Oct 30 19:21:06.860330 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:06.860302    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c9fc077-3f23-40d1-ba6e-2575551e01a8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fw65nx\" (UID: \"0c9fc077-3f23-40d1-ba6e-2575551e01a8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fw65nx"
Oct 30 19:21:06.860507 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:06.860334    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c9fc077-3f23-40d1-ba6e-2575551e01a8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fw65nx\" (UID: \"0c9fc077-3f23-40d1-ba6e-2575551e01a8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fw65nx"
Oct 30 19:21:06.860507 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:06.860374    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9fc077-3f23-40d1-ba6e-2575551e01a8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fw65nx\" (UID: \"0c9fc077-3f23-40d1-ba6e-2575551e01a8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fw65nx"
Oct 30 19:21:06.860507 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:06.860459    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c9fc077-3f23-40d1-ba6e-2575551e01a8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fw65nx\" (UID: \"0c9fc077-3f23-40d1-ba6e-2575551e01a8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fw65nx"
Oct 30 19:21:06.860507 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:06.860487    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9fc077-3f23-40d1-ba6e-2575551e01a8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fw65nx\" (UID: \"0c9fc077-3f23-40d1-ba6e-2575551e01a8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fw65nx"
Oct 30 19:21:06.860656 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:06.860528    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm5c6\" (UniqueName: \"kubernetes.io/projected/0c9fc077-3f23-40d1-ba6e-2575551e01a8-kube-api-access-fm5c6\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fw65nx\" (UID: \"0c9fc077-3f23-40d1-ba6e-2575551e01a8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fw65nx"
Oct 30 19:21:06.961986 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:06.961950    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c9fc077-3f23-40d1-ba6e-2575551e01a8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fw65nx\" (UID: \"0c9fc077-3f23-40d1-ba6e-2575551e01a8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fw65nx"
Oct 30 19:21:06.962182 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:06.961991    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c9fc077-3f23-40d1-ba6e-2575551e01a8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fw65nx\" (UID: \"0c9fc077-3f23-40d1-ba6e-2575551e01a8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fw65nx"
Oct 30 19:21:06.962182 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:06.962026    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c9fc077-3f23-40d1-ba6e-2575551e01a8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fw65nx\" (UID: \"0c9fc077-3f23-40d1-ba6e-2575551e01a8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fw65nx"
Oct 30 19:21:06.962182 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:06.962076    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c9fc077-3f23-40d1-ba6e-2575551e01a8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fw65nx\" (UID: \"0c9fc077-3f23-40d1-ba6e-2575551e01a8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fw65nx"
Oct 30 19:21:06.962182 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:06.962106    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c9fc077-3f23-40d1-ba6e-2575551e01a8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fw65nx\" (UID: \"0c9fc077-3f23-40d1-ba6e-2575551e01a8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fw65nx"
Oct 30 19:21:06.962413 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:06.962215    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9fc077-3f23-40d1-ba6e-2575551e01a8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fw65nx\" (UID: \"0c9fc077-3f23-40d1-ba6e-2575551e01a8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fw65nx"
Oct 30 19:21:06.962413 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:06.962282    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c9fc077-3f23-40d1-ba6e-2575551e01a8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fw65nx\" (UID: \"0c9fc077-3f23-40d1-ba6e-2575551e01a8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fw65nx"
Oct 30 19:21:06.962413 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:06.962303    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9fc077-3f23-40d1-ba6e-2575551e01a8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fw65nx\" (UID: \"0c9fc077-3f23-40d1-ba6e-2575551e01a8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fw65nx"
Oct 30 19:21:06.962413 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:06.962404    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm5c6\" (UniqueName: \"kubernetes.io/projected/0c9fc077-3f23-40d1-ba6e-2575551e01a8-kube-api-access-fm5c6\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fw65nx\" (UID: \"0c9fc077-3f23-40d1-ba6e-2575551e01a8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fw65nx"
Oct 30 19:21:06.962629 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:06.962560    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c9fc077-3f23-40d1-ba6e-2575551e01a8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fw65nx\" (UID: \"0c9fc077-3f23-40d1-ba6e-2575551e01a8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fw65nx"
Oct 30 19:21:06.962686 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:06.962650    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9fc077-3f23-40d1-ba6e-2575551e01a8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fw65nx\" (UID: \"0c9fc077-3f23-40d1-ba6e-2575551e01a8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fw65nx"
Oct 30 19:21:06.962890 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:06.962842    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c9fc077-3f23-40d1-ba6e-2575551e01a8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fw65nx\" (UID: \"0c9fc077-3f23-40d1-ba6e-2575551e01a8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fw65nx"
Oct 30 19:21:06.963009 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:06.962951    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c9fc077-3f23-40d1-ba6e-2575551e01a8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fw65nx\" (UID: \"0c9fc077-3f23-40d1-ba6e-2575551e01a8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fw65nx"
Oct 30 19:21:06.963009 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:06.963000    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9fc077-3f23-40d1-ba6e-2575551e01a8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fw65nx\" (UID: \"0c9fc077-3f23-40d1-ba6e-2575551e01a8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fw65nx"
Oct 30 19:21:06.965243 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:06.965216    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c9fc077-3f23-40d1-ba6e-2575551e01a8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fw65nx\" (UID: \"0c9fc077-3f23-40d1-ba6e-2575551e01a8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fw65nx"
Oct 30 19:21:06.965388 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:06.965365    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c9fc077-3f23-40d1-ba6e-2575551e01a8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fw65nx\" (UID: \"0c9fc077-3f23-40d1-ba6e-2575551e01a8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fw65nx"
Oct 30 19:21:06.973028 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:06.973001    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm5c6\" (UniqueName: \"kubernetes.io/projected/0c9fc077-3f23-40d1-ba6e-2575551e01a8-kube-api-access-fm5c6\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fw65nx\" (UID: \"0c9fc077-3f23-40d1-ba6e-2575551e01a8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fw65nx"
Oct 30 19:21:06.973194 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:06.973036    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c9fc077-3f23-40d1-ba6e-2575551e01a8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fw65nx\" (UID: \"0c9fc077-3f23-40d1-ba6e-2575551e01a8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fw65nx"
Oct 30 19:21:07.109562 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:07.109473    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fw65nx"
Oct 30 19:21:07.397155 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:07.397128    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fw65nx"]
Oct 30 19:21:07.399848 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:21:07.399817    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c9fc077_3f23_40d1_ba6e_2575551e01a8.slice/crio-731bb2423e320541124295a460e2fef53256a79468028d54ffc263f3ae7fceb6 WatchSource:0}: Error finding container 731bb2423e320541124295a460e2fef53256a79468028d54ffc263f3ae7fceb6: Status 404 returned error can't find the container with id 731bb2423e320541124295a460e2fef53256a79468028d54ffc263f3ae7fceb6
Oct 30 19:21:07.402214 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:07.402173    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:21:07.402310 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:07.402264    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:21:07.402351 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:07.402305    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:21:08.095589 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:08.095539    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fw65nx" event={"ID":"0c9fc077-3f23-40d1-ba6e-2575551e01a8","Type":"ContainerStarted","Data":"2af745e857761e56c0767a91df219a0425b75424931d91bd894fd066df13afba"}
Oct 30 19:21:08.096074 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:08.095605    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fw65nx" event={"ID":"0c9fc077-3f23-40d1-ba6e-2575551e01a8","Type":"ContainerStarted","Data":"731bb2423e320541124295a460e2fef53256a79468028d54ffc263f3ae7fceb6"}
Oct 30 19:21:08.110620 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:08.110592    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fw65nx"
Oct 30 19:21:08.115454 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:08.115431    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fw65nx"
Oct 30 19:21:08.123446 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:08.123401    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fw65nx" podStartSLOduration=2.123389038 podStartE2EDuration="2.123389038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:08.121998732 +0000 UTC m=+287.633058698" watchObservedRunningTime="2026-10-30 19:21:08.123389038 +0000 UTC m=+287.634449000"
Oct 30 19:21:08.558669 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:08.558630    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-r7n77"]
Oct 30 19:21:08.562210 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:08.562190    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-r7n77"
Oct 30 19:21:08.574477 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:08.574457    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2682b\""
Oct 30 19:21:08.584910 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:08.584886    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-r7n77"]
Oct 30 19:21:08.679513 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:08.679480    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x78tt\" (UniqueName: \"kubernetes.io/projected/e276e7d0-9135-41ac-8036-374dc1061970-kube-api-access-x78tt\") pod \"authorino-57f7f8659f-r7n77\" (UID: \"e276e7d0-9135-41ac-8036-374dc1061970\") " pod="kuadrant-system/authorino-57f7f8659f-r7n77"
Oct 30 19:21:08.780598 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:08.780561    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x78tt\" (UniqueName: \"kubernetes.io/projected/e276e7d0-9135-41ac-8036-374dc1061970-kube-api-access-x78tt\") pod \"authorino-57f7f8659f-r7n77\" (UID: \"e276e7d0-9135-41ac-8036-374dc1061970\") " pod="kuadrant-system/authorino-57f7f8659f-r7n77"
Oct 30 19:21:08.814338 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:08.814265    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x78tt\" (UniqueName: \"kubernetes.io/projected/e276e7d0-9135-41ac-8036-374dc1061970-kube-api-access-x78tt\") pod \"authorino-57f7f8659f-r7n77\" (UID: \"e276e7d0-9135-41ac-8036-374dc1061970\") " pod="kuadrant-system/authorino-57f7f8659f-r7n77"
Oct 30 19:21:08.884993 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:08.884963    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-r7n77"
Oct 30 19:21:09.047568 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:09.047538    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-9lkpq"
Oct 30 19:21:09.071467 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:09.071391    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-c74vl"]
Oct 30 19:21:09.076146 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:09.076122    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-c74vl"
Oct 30 19:21:09.090943 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:09.090918    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-c74vl"]
Oct 30 19:21:09.100583 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:09.100391    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fw65nx"
Oct 30 19:21:09.101790 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:09.101772    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fw65nx"
Oct 30 19:21:09.169766 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:09.169715    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-r7n77"]
Oct 30 19:21:09.173893 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:21:09.173866    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode276e7d0_9135_41ac_8036_374dc1061970.slice/crio-48664232642a7bfafc3c222e229beaf693e24e0ac4bf72602b774d250eb73aa3 WatchSource:0}: Error finding container 48664232642a7bfafc3c222e229beaf693e24e0ac4bf72602b774d250eb73aa3: Status 404 returned error can't find the container with id 48664232642a7bfafc3c222e229beaf693e24e0ac4bf72602b774d250eb73aa3
Oct 30 19:21:09.182860 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:09.182835    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk4dq\" (UniqueName: \"kubernetes.io/projected/09d9aa32-f70e-4e76-9316-6f58de2a65cc-kube-api-access-sk4dq\") pod \"authorino-56c6c9ddd5-c74vl\" (UID: \"09d9aa32-f70e-4e76-9316-6f58de2a65cc\") " pod="kuadrant-system/authorino-56c6c9ddd5-c74vl"
Oct 30 19:21:09.283402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:09.283364    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk4dq\" (UniqueName: \"kubernetes.io/projected/09d9aa32-f70e-4e76-9316-6f58de2a65cc-kube-api-access-sk4dq\") pod \"authorino-56c6c9ddd5-c74vl\" (UID: \"09d9aa32-f70e-4e76-9316-6f58de2a65cc\") " pod="kuadrant-system/authorino-56c6c9ddd5-c74vl"
Oct 30 19:21:09.296590 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:09.296565    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk4dq\" (UniqueName: \"kubernetes.io/projected/09d9aa32-f70e-4e76-9316-6f58de2a65cc-kube-api-access-sk4dq\") pod \"authorino-56c6c9ddd5-c74vl\" (UID: \"09d9aa32-f70e-4e76-9316-6f58de2a65cc\") " pod="kuadrant-system/authorino-56c6c9ddd5-c74vl"
Oct 30 19:21:09.392375 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:09.392296    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-c74vl"
Oct 30 19:21:09.668193 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:09.668164    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-c74vl"]
Oct 30 19:21:09.669990 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:21:09.669965    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09d9aa32_f70e_4e76_9316_6f58de2a65cc.slice/crio-ec04c5db2dae560031193fc7b376972ee391095e3b44ba6916c1dd2d46644458 WatchSource:0}: Error finding container ec04c5db2dae560031193fc7b376972ee391095e3b44ba6916c1dd2d46644458: Status 404 returned error can't find the container with id ec04c5db2dae560031193fc7b376972ee391095e3b44ba6916c1dd2d46644458
Oct 30 19:21:10.105319 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:10.105273    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-r7n77" event={"ID":"e276e7d0-9135-41ac-8036-374dc1061970","Type":"ContainerStarted","Data":"48664232642a7bfafc3c222e229beaf693e24e0ac4bf72602b774d250eb73aa3"}
Oct 30 19:21:10.106409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:10.106389    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-c74vl" event={"ID":"09d9aa32-f70e-4e76-9316-6f58de2a65cc","Type":"ContainerStarted","Data":"ec04c5db2dae560031193fc7b376972ee391095e3b44ba6916c1dd2d46644458"}
Oct 30 19:21:12.119562 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:12.119524    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-r7n77" event={"ID":"e276e7d0-9135-41ac-8036-374dc1061970","Type":"ContainerStarted","Data":"55f6384b8aeec631092867308f7289e3e4abd245cb3ea9a3dff43730811cd66c"}
Oct 30 19:21:12.120963 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:12.120936    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-c74vl" event={"ID":"09d9aa32-f70e-4e76-9316-6f58de2a65cc","Type":"ContainerStarted","Data":"b70841909ccec8da9105fd3f4d6c5598f3f16f3dbc896e05b8688195b556556b"}
Oct 30 19:21:12.136148 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:12.136109    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-r7n77" podStartSLOduration=1.726599371 podStartE2EDuration="4.136095771s" totalImagesPullingTime="2.4094964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:12.135411833 +0000 UTC m=+291.646471798" watchObservedRunningTime="2026-10-30 19:21:12.136095771 +0000 UTC m=+291.647155735"
Oct 30 19:21:12.151832 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:12.151777    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-c74vl" podStartSLOduration=1.283321498 podStartE2EDuration="3.151759792s" totalImagesPullingTime="1.868438294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:12.151429852 +0000 UTC m=+291.662489819" watchObservedRunningTime="2026-10-30 19:21:12.151759792 +0000 UTC m=+291.662819761"
Oct 30 19:21:12.178687 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:12.178660    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-r7n77"]
Oct 30 19:21:13.547902 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:13.547869    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-56f87bd649-zp4kd"]
Oct 30 19:21:13.551436 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:13.551421    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-56f87bd649-zp4kd"
Oct 30 19:21:13.556681 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:13.556661    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Oct 30 19:21:13.556834 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:13.556723    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:21:13.556834 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:13.556791    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 30 19:21:13.556834 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:13.556824    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Oct 30 19:21:13.557002 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:13.556912    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Oct 30 19:21:13.557050 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:13.557008    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-vhxgj\""
Oct 30 19:21:13.562356 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:13.562338    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-56f87bd649-zp4kd"]
Oct 30 19:21:13.624112 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:13.624082    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d20b0117-df8b-4662-9a06-28868dba0801-manager-config\") pod \"jobset-controller-manager-56f87bd649-zp4kd\" (UID: \"d20b0117-df8b-4662-9a06-28868dba0801\") " pod="openshift-jobset-operator/jobset-controller-manager-56f87bd649-zp4kd"
Oct 30 19:21:13.624301 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:13.624149    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d20b0117-df8b-4662-9a06-28868dba0801-cert\") pod \"jobset-controller-manager-56f87bd649-zp4kd\" (UID: \"d20b0117-df8b-4662-9a06-28868dba0801\") " pod="openshift-jobset-operator/jobset-controller-manager-56f87bd649-zp4kd"
Oct 30 19:21:13.624301 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:13.624242    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz5kh\" (UniqueName: \"kubernetes.io/projected/d20b0117-df8b-4662-9a06-28868dba0801-kube-api-access-mz5kh\") pod \"jobset-controller-manager-56f87bd649-zp4kd\" (UID: \"d20b0117-df8b-4662-9a06-28868dba0801\") " pod="openshift-jobset-operator/jobset-controller-manager-56f87bd649-zp4kd"
Oct 30 19:21:13.624301 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:13.624285    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d20b0117-df8b-4662-9a06-28868dba0801-metrics-certs\") pod \"jobset-controller-manager-56f87bd649-zp4kd\" (UID: \"d20b0117-df8b-4662-9a06-28868dba0801\") " pod="openshift-jobset-operator/jobset-controller-manager-56f87bd649-zp4kd"
Oct 30 19:21:13.725083 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:13.725054    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d20b0117-df8b-4662-9a06-28868dba0801-cert\") pod \"jobset-controller-manager-56f87bd649-zp4kd\" (UID: \"d20b0117-df8b-4662-9a06-28868dba0801\") " pod="openshift-jobset-operator/jobset-controller-manager-56f87bd649-zp4kd"
Oct 30 19:21:13.725234 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:13.725095    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz5kh\" (UniqueName: \"kubernetes.io/projected/d20b0117-df8b-4662-9a06-28868dba0801-kube-api-access-mz5kh\") pod \"jobset-controller-manager-56f87bd649-zp4kd\" (UID: \"d20b0117-df8b-4662-9a06-28868dba0801\") " pod="openshift-jobset-operator/jobset-controller-manager-56f87bd649-zp4kd"
Oct 30 19:21:13.725234 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:13.725118    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d20b0117-df8b-4662-9a06-28868dba0801-metrics-certs\") pod \"jobset-controller-manager-56f87bd649-zp4kd\" (UID: \"d20b0117-df8b-4662-9a06-28868dba0801\") " pod="openshift-jobset-operator/jobset-controller-manager-56f87bd649-zp4kd"
Oct 30 19:21:13.725234 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:13.725157    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d20b0117-df8b-4662-9a06-28868dba0801-manager-config\") pod \"jobset-controller-manager-56f87bd649-zp4kd\" (UID: \"d20b0117-df8b-4662-9a06-28868dba0801\") " pod="openshift-jobset-operator/jobset-controller-manager-56f87bd649-zp4kd"
Oct 30 19:21:13.725820 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:13.725799    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d20b0117-df8b-4662-9a06-28868dba0801-manager-config\") pod \"jobset-controller-manager-56f87bd649-zp4kd\" (UID: \"d20b0117-df8b-4662-9a06-28868dba0801\") " pod="openshift-jobset-operator/jobset-controller-manager-56f87bd649-zp4kd"
Oct 30 19:21:13.727874 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:13.727850    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d20b0117-df8b-4662-9a06-28868dba0801-metrics-certs\") pod \"jobset-controller-manager-56f87bd649-zp4kd\" (UID: \"d20b0117-df8b-4662-9a06-28868dba0801\") " pod="openshift-jobset-operator/jobset-controller-manager-56f87bd649-zp4kd"
Oct 30 19:21:13.727957 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:13.727859    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d20b0117-df8b-4662-9a06-28868dba0801-cert\") pod \"jobset-controller-manager-56f87bd649-zp4kd\" (UID: \"d20b0117-df8b-4662-9a06-28868dba0801\") " pod="openshift-jobset-operator/jobset-controller-manager-56f87bd649-zp4kd"
Oct 30 19:21:13.733212 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:13.733189    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz5kh\" (UniqueName: \"kubernetes.io/projected/d20b0117-df8b-4662-9a06-28868dba0801-kube-api-access-mz5kh\") pod \"jobset-controller-manager-56f87bd649-zp4kd\" (UID: \"d20b0117-df8b-4662-9a06-28868dba0801\") " pod="openshift-jobset-operator/jobset-controller-manager-56f87bd649-zp4kd"
Oct 30 19:21:13.863515 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:13.863433    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-56f87bd649-zp4kd"
Oct 30 19:21:14.131893 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:14.131801    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-r7n77" podUID="e276e7d0-9135-41ac-8036-374dc1061970" containerName="authorino" containerID="cri-o://55f6384b8aeec631092867308f7289e3e4abd245cb3ea9a3dff43730811cd66c" gracePeriod=30
Oct 30 19:21:14.150196 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:14.150178    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-56f87bd649-zp4kd"]
Oct 30 19:21:14.169543 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:21:14.169517    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd20b0117_df8b_4662_9a06_28868dba0801.slice/crio-6625033a21d315dba9d621a32f382eac91fd176dbe49f6878915d1ea617011e6 WatchSource:0}: Error finding container 6625033a21d315dba9d621a32f382eac91fd176dbe49f6878915d1ea617011e6: Status 404 returned error can't find the container with id 6625033a21d315dba9d621a32f382eac91fd176dbe49f6878915d1ea617011e6
Oct 30 19:21:14.367422 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:14.367397    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-r7n77"
Oct 30 19:21:14.432877 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:14.432802    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e276e7d0-9135-41ac-8036-374dc1061970-kube-api-access-x78tt\" (UniqueName: \"kubernetes.io/projected/e276e7d0-9135-41ac-8036-374dc1061970-kube-api-access-x78tt\") pod \"e276e7d0-9135-41ac-8036-374dc1061970\" (UID: \"e276e7d0-9135-41ac-8036-374dc1061970\") "
Oct 30 19:21:14.435118 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:14.435094    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e276e7d0-9135-41ac-8036-374dc1061970-kube-api-access-x78tt" pod "e276e7d0-9135-41ac-8036-374dc1061970" (UID: "e276e7d0-9135-41ac-8036-374dc1061970"). InnerVolumeSpecName "kube-api-access-x78tt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:14.534234 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:14.534206    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x78tt\" (UniqueName: \"kubernetes.io/projected/e276e7d0-9135-41ac-8036-374dc1061970-kube-api-access-x78tt\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:15.137680 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:15.137597    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-56f87bd649-zp4kd" event={"ID":"d20b0117-df8b-4662-9a06-28868dba0801","Type":"ContainerStarted","Data":"6625033a21d315dba9d621a32f382eac91fd176dbe49f6878915d1ea617011e6"}
Oct 30 19:21:15.138876 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:15.138853    5043 generic.go:356] "Generic (PLEG): container finished" podID="e276e7d0-9135-41ac-8036-374dc1061970" containerID="55f6384b8aeec631092867308f7289e3e4abd245cb3ea9a3dff43730811cd66c" exitCode=0
Oct 30 19:21:15.138988 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:15.138895    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-r7n77"
Oct 30 19:21:15.138988 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:15.138920    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-r7n77" event={"ID":"e276e7d0-9135-41ac-8036-374dc1061970","Type":"ContainerDied","Data":"55f6384b8aeec631092867308f7289e3e4abd245cb3ea9a3dff43730811cd66c"}
Oct 30 19:21:15.138988 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:15.138944    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-r7n77" event={"ID":"e276e7d0-9135-41ac-8036-374dc1061970","Type":"ContainerDied","Data":"48664232642a7bfafc3c222e229beaf693e24e0ac4bf72602b774d250eb73aa3"}
Oct 30 19:21:15.138988 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:15.138959    5043 scope.go:122] "RemoveContainer" containerID="55f6384b8aeec631092867308f7289e3e4abd245cb3ea9a3dff43730811cd66c"
Oct 30 19:21:15.147607 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:15.147589    5043 scope.go:122] "RemoveContainer" containerID="55f6384b8aeec631092867308f7289e3e4abd245cb3ea9a3dff43730811cd66c"
Oct 30 19:21:15.147981 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:21:15.147962    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55f6384b8aeec631092867308f7289e3e4abd245cb3ea9a3dff43730811cd66c\": container with ID starting with 55f6384b8aeec631092867308f7289e3e4abd245cb3ea9a3dff43730811cd66c not found: ID does not exist" containerID="55f6384b8aeec631092867308f7289e3e4abd245cb3ea9a3dff43730811cd66c"
Oct 30 19:21:15.148054 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:15.147990    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55f6384b8aeec631092867308f7289e3e4abd245cb3ea9a3dff43730811cd66c"} err="failed to get container status \"55f6384b8aeec631092867308f7289e3e4abd245cb3ea9a3dff43730811cd66c\": rpc error: code = NotFound desc = could not find container \"55f6384b8aeec631092867308f7289e3e4abd245cb3ea9a3dff43730811cd66c\": container with ID starting with 55f6384b8aeec631092867308f7289e3e4abd245cb3ea9a3dff43730811cd66c not found: ID does not exist"
Oct 30 19:21:15.157593 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:15.157572    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-r7n77"]
Oct 30 19:21:15.162391 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:15.162372    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-r7n77"]
Oct 30 19:21:16.085971 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:16.085940    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sdss4"
Oct 30 19:21:16.643794 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:16.643762    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e276e7d0-9135-41ac-8036-374dc1061970" path="/var/lib/kubelet/pods/e276e7d0-9135-41ac-8036-374dc1061970/volumes"
Oct 30 19:21:17.152319 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:17.152279    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-56f87bd649-zp4kd" event={"ID":"d20b0117-df8b-4662-9a06-28868dba0801","Type":"ContainerStarted","Data":"88b59b7872d4c2f98c5f3d71b014ccc61e1a43937c4432b2ef845e2b888268ee"}
Oct 30 19:21:17.152551 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:17.152534    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-56f87bd649-zp4kd"
Oct 30 19:21:17.312676 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:17.312558    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-56f87bd649-zp4kd" podStartSLOduration=1.5585651440000001 podStartE2EDuration="4.312541496s" totalImagesPullingTime="2.753976352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:17.312009381 +0000 UTC m=+296.823069345" watchObservedRunningTime="2026-10-30 19:21:17.312541496 +0000 UTC m=+296.823601461"
Oct 30 19:21:20.664598 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:20.664573    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cmw6w_76e21916-0744-4499-a0cb-64a7a44a8285/console-operator/2.log"
Oct 30 19:21:20.664972 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:20.664581    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cmw6w_76e21916-0744-4499-a0cb-64a7a44a8285/console-operator/2.log"
Oct 30 19:21:24.976587 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:21:24.976550    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:21:28.163526 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:21:28.163496    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-56f87bd649-zp4kd"
Oct 30 19:21:54.974534 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:21:54.974499    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:22:24.975192 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:22:24.975156    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:22:36.429112 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:36.429078    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-svtcp"]
Oct 30 19:22:36.429498 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:36.429477    5043 cpu_manager.go:414] "containerMap: removing container" podUID="e276e7d0-9135-41ac-8036-374dc1061970" containerName="authorino"
Oct 30 19:22:36.429498 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:36.429490    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e276e7d0-9135-41ac-8036-374dc1061970" containerName="authorino"
Oct 30 19:22:36.429602 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:36.429591    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="e276e7d0-9135-41ac-8036-374dc1061970" containerName="authorino"
Oct 30 19:22:36.432580 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:36.432554    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-svtcp"
Oct 30 19:22:36.444011 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:36.443982    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-svtcp"]
Oct 30 19:22:36.550655 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:36.550619    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7d2f\" (UniqueName: \"kubernetes.io/projected/bb5be83f-c925-44f6-94a6-f76b44740d3c-kube-api-access-p7d2f\") pod \"authorino-6f6944bcdd-svtcp\" (UID: \"bb5be83f-c925-44f6-94a6-f76b44740d3c\") " pod="kuadrant-system/authorino-6f6944bcdd-svtcp"
Oct 30 19:22:36.630830 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:36.630796    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-svtcp"]
Oct 30 19:22:36.631034 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:22:36.631012    5043 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-p7d2f], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-svtcp" podUID="bb5be83f-c925-44f6-94a6-f76b44740d3c"
Oct 30 19:22:36.651686 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:36.651659    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7d2f\" (UniqueName: \"kubernetes.io/projected/bb5be83f-c925-44f6-94a6-f76b44740d3c-kube-api-access-p7d2f\") pod \"authorino-6f6944bcdd-svtcp\" (UID: \"bb5be83f-c925-44f6-94a6-f76b44740d3c\") " pod="kuadrant-system/authorino-6f6944bcdd-svtcp"
Oct 30 19:22:36.658026 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:36.658003    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-q5jmf"]
Oct 30 19:22:36.661463 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:36.661443    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-q5jmf"
Oct 30 19:22:36.662630 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:36.662599    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7d2f\" (UniqueName: \"kubernetes.io/projected/bb5be83f-c925-44f6-94a6-f76b44740d3c-kube-api-access-p7d2f\") pod \"authorino-6f6944bcdd-svtcp\" (UID: \"bb5be83f-c925-44f6-94a6-f76b44740d3c\") " pod="kuadrant-system/authorino-6f6944bcdd-svtcp"
Oct 30 19:22:36.668249 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:36.668227    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-q5jmf"]
Oct 30 19:22:36.738395 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:36.738370    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 30 19:22:36.752802 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:36.752780    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hngbb\" (UniqueName: \"kubernetes.io/projected/3d99fadf-e26f-4611-bc42-7c795a44ee56-kube-api-access-hngbb\") pod \"authorino-6889cbd8c4-q5jmf\" (UID: \"3d99fadf-e26f-4611-bc42-7c795a44ee56\") " pod="kuadrant-system/authorino-6889cbd8c4-q5jmf"
Oct 30 19:22:36.752921 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:36.752861    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3d99fadf-e26f-4611-bc42-7c795a44ee56-tls-cert\") pod \"authorino-6889cbd8c4-q5jmf\" (UID: \"3d99fadf-e26f-4611-bc42-7c795a44ee56\") " pod="kuadrant-system/authorino-6889cbd8c4-q5jmf"
Oct 30 19:22:36.792125 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:36.792099    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-q5jmf"]
Oct 30 19:22:36.792325 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:22:36.792308    5043 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-hngbb tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-q5jmf" podUID="3d99fadf-e26f-4611-bc42-7c795a44ee56"
Oct 30 19:22:36.823989 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:36.823959    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6cbbd8b8bb-czqvm"]
Oct 30 19:22:36.827526 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:36.827508    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cbbd8b8bb-czqvm"
Oct 30 19:22:36.837581 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:36.837554    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6cbbd8b8bb-czqvm"]
Oct 30 19:22:36.853824 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:36.853790    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3d99fadf-e26f-4611-bc42-7c795a44ee56-tls-cert\") pod \"authorino-6889cbd8c4-q5jmf\" (UID: \"3d99fadf-e26f-4611-bc42-7c795a44ee56\") " pod="kuadrant-system/authorino-6889cbd8c4-q5jmf"
Oct 30 19:22:36.853994 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:36.853853    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hngbb\" (UniqueName: \"kubernetes.io/projected/3d99fadf-e26f-4611-bc42-7c795a44ee56-kube-api-access-hngbb\") pod \"authorino-6889cbd8c4-q5jmf\" (UID: \"3d99fadf-e26f-4611-bc42-7c795a44ee56\") " pod="kuadrant-system/authorino-6889cbd8c4-q5jmf"
Oct 30 19:22:36.856495 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:36.856469    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3d99fadf-e26f-4611-bc42-7c795a44ee56-tls-cert\") pod \"authorino-6889cbd8c4-q5jmf\" (UID: \"3d99fadf-e26f-4611-bc42-7c795a44ee56\") " pod="kuadrant-system/authorino-6889cbd8c4-q5jmf"
Oct 30 19:22:36.861065 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:36.861041    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hngbb\" (UniqueName: \"kubernetes.io/projected/3d99fadf-e26f-4611-bc42-7c795a44ee56-kube-api-access-hngbb\") pod \"authorino-6889cbd8c4-q5jmf\" (UID: \"3d99fadf-e26f-4611-bc42-7c795a44ee56\") " pod="kuadrant-system/authorino-6889cbd8c4-q5jmf"
Oct 30 19:22:36.955088 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:36.955059    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64zk9\" (UniqueName: \"kubernetes.io/projected/69ffd927-711e-460a-94d3-b71b1c51bfce-kube-api-access-64zk9\") pod \"authorino-6cbbd8b8bb-czqvm\" (UID: \"69ffd927-711e-460a-94d3-b71b1c51bfce\") " pod="kuadrant-system/authorino-6cbbd8b8bb-czqvm"
Oct 30 19:22:36.955264 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:36.955129    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/69ffd927-711e-460a-94d3-b71b1c51bfce-tls-cert\") pod \"authorino-6cbbd8b8bb-czqvm\" (UID: \"69ffd927-711e-460a-94d3-b71b1c51bfce\") " pod="kuadrant-system/authorino-6cbbd8b8bb-czqvm"
Oct 30 19:22:37.056658 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:37.056566    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64zk9\" (UniqueName: \"kubernetes.io/projected/69ffd927-711e-460a-94d3-b71b1c51bfce-kube-api-access-64zk9\") pod \"authorino-6cbbd8b8bb-czqvm\" (UID: \"69ffd927-711e-460a-94d3-b71b1c51bfce\") " pod="kuadrant-system/authorino-6cbbd8b8bb-czqvm"
Oct 30 19:22:37.056869 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:37.056683    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/69ffd927-711e-460a-94d3-b71b1c51bfce-tls-cert\") pod \"authorino-6cbbd8b8bb-czqvm\" (UID: \"69ffd927-711e-460a-94d3-b71b1c51bfce\") " pod="kuadrant-system/authorino-6cbbd8b8bb-czqvm"
Oct 30 19:22:37.059355 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:37.059325    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/69ffd927-711e-460a-94d3-b71b1c51bfce-tls-cert\") pod \"authorino-6cbbd8b8bb-czqvm\" (UID: \"69ffd927-711e-460a-94d3-b71b1c51bfce\") " pod="kuadrant-system/authorino-6cbbd8b8bb-czqvm"
Oct 30 19:22:37.065237 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:37.065212    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64zk9\" (UniqueName: \"kubernetes.io/projected/69ffd927-711e-460a-94d3-b71b1c51bfce-kube-api-access-64zk9\") pod \"authorino-6cbbd8b8bb-czqvm\" (UID: \"69ffd927-711e-460a-94d3-b71b1c51bfce\") " pod="kuadrant-system/authorino-6cbbd8b8bb-czqvm"
Oct 30 19:22:37.139773 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:37.139717    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cbbd8b8bb-czqvm"
Oct 30 19:22:37.411681 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:37.411655    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6cbbd8b8bb-czqvm"]
Oct 30 19:22:37.413641 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:22:37.413610    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69ffd927_711e_460a_94d3_b71b1c51bfce.slice/crio-d8865b8d98864f610476202a2b21b26cbca5a2a66f9cf78b7b5abe7c475bfca6 WatchSource:0}: Error finding container d8865b8d98864f610476202a2b21b26cbca5a2a66f9cf78b7b5abe7c475bfca6: Status 404 returned error can't find the container with id d8865b8d98864f610476202a2b21b26cbca5a2a66f9cf78b7b5abe7c475bfca6
Oct 30 19:22:37.414930 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:37.414911    5043 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:22:37.534660 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:37.534621    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cbbd8b8bb-czqvm" event={"ID":"69ffd927-711e-460a-94d3-b71b1c51bfce","Type":"ContainerStarted","Data":"d8865b8d98864f610476202a2b21b26cbca5a2a66f9cf78b7b5abe7c475bfca6"}
Oct 30 19:22:37.535146 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:37.534703    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-q5jmf"
Oct 30 19:22:37.535146 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:37.534858    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-svtcp"
Oct 30 19:22:37.543144 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:37.543116    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-q5jmf"
Oct 30 19:22:37.545144 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:37.545129    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-svtcp"
Oct 30 19:22:37.663076 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:37.662995    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d99fadf-e26f-4611-bc42-7c795a44ee56-kube-api-access-hngbb\" (UniqueName: \"kubernetes.io/projected/3d99fadf-e26f-4611-bc42-7c795a44ee56-kube-api-access-hngbb\") pod \"3d99fadf-e26f-4611-bc42-7c795a44ee56\" (UID: \"3d99fadf-e26f-4611-bc42-7c795a44ee56\") "
Oct 30 19:22:37.663076 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:37.663050    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb5be83f-c925-44f6-94a6-f76b44740d3c-kube-api-access-p7d2f\" (UniqueName: \"kubernetes.io/projected/bb5be83f-c925-44f6-94a6-f76b44740d3c-kube-api-access-p7d2f\") pod \"bb5be83f-c925-44f6-94a6-f76b44740d3c\" (UID: \"bb5be83f-c925-44f6-94a6-f76b44740d3c\") "
Oct 30 19:22:37.663275 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:37.663082    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d99fadf-e26f-4611-bc42-7c795a44ee56-tls-cert\" (UniqueName: \"kubernetes.io/secret/3d99fadf-e26f-4611-bc42-7c795a44ee56-tls-cert\") pod \"3d99fadf-e26f-4611-bc42-7c795a44ee56\" (UID: \"3d99fadf-e26f-4611-bc42-7c795a44ee56\") "
Oct 30 19:22:37.665474 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:37.665438    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb5be83f-c925-44f6-94a6-f76b44740d3c-kube-api-access-p7d2f" pod "bb5be83f-c925-44f6-94a6-f76b44740d3c" (UID: "bb5be83f-c925-44f6-94a6-f76b44740d3c"). InnerVolumeSpecName "kube-api-access-p7d2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:37.665583 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:37.665443    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d99fadf-e26f-4611-bc42-7c795a44ee56-tls-cert" pod "3d99fadf-e26f-4611-bc42-7c795a44ee56" (UID: "3d99fadf-e26f-4611-bc42-7c795a44ee56"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:22:37.665583 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:37.665450    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d99fadf-e26f-4611-bc42-7c795a44ee56-kube-api-access-hngbb" pod "3d99fadf-e26f-4611-bc42-7c795a44ee56" (UID: "3d99fadf-e26f-4611-bc42-7c795a44ee56"). InnerVolumeSpecName "kube-api-access-hngbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:37.763907 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:37.763870    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hngbb\" (UniqueName: \"kubernetes.io/projected/3d99fadf-e26f-4611-bc42-7c795a44ee56-kube-api-access-hngbb\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:37.763907 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:37.763901    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7d2f\" (UniqueName: \"kubernetes.io/projected/bb5be83f-c925-44f6-94a6-f76b44740d3c-kube-api-access-p7d2f\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:37.763907 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:37.763911    5043 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3d99fadf-e26f-4611-bc42-7c795a44ee56-tls-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:38.539527 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:38.539494    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-svtcp"
Oct 30 19:22:38.539527 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:38.539526    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-q5jmf"
Oct 30 19:22:38.572353 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:38.572309    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-q5jmf"]
Oct 30 19:22:38.579852 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:38.579823    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-q5jmf"]
Oct 30 19:22:38.607161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:38.607134    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-svtcp"]
Oct 30 19:22:38.622259 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:38.622232    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-svtcp"]
Oct 30 19:22:38.643951 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:38.643923    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d99fadf-e26f-4611-bc42-7c795a44ee56" path="/var/lib/kubelet/pods/3d99fadf-e26f-4611-bc42-7c795a44ee56/volumes"
Oct 30 19:22:38.644229 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:38.644206    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb5be83f-c925-44f6-94a6-f76b44740d3c" path="/var/lib/kubelet/pods/bb5be83f-c925-44f6-94a6-f76b44740d3c/volumes"
Oct 30 19:22:39.545398 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:39.545354    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cbbd8b8bb-czqvm" event={"ID":"69ffd927-711e-460a-94d3-b71b1c51bfce","Type":"ContainerStarted","Data":"823a2431e599f839dd9129ae5596fd52b6a4507f475765f8ceee2ad923bf11f5"}
Oct 30 19:22:39.589084 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:39.589004    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6cbbd8b8bb-czqvm" podStartSLOduration=2.45777049 podStartE2EDuration="3.588988565s" totalImagesPullingTime="1.131218075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:39.588269456 +0000 UTC m=+379.099329423" watchObservedRunningTime="2026-10-30 19:22:39.588988565 +0000 UTC m=+379.100048529"
Oct 30 19:22:39.628696 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:39.628657    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-c74vl"]
Oct 30 19:22:39.629109 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:39.628912    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-c74vl" podUID="09d9aa32-f70e-4e76-9316-6f58de2a65cc" containerName="authorino" containerID="cri-o://b70841909ccec8da9105fd3f4d6c5598f3f16f3dbc896e05b8688195b556556b" gracePeriod=30
Oct 30 19:22:39.915053 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:39.915030    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-c74vl"
Oct 30 19:22:40.086055 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:40.085968    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09d9aa32-f70e-4e76-9316-6f58de2a65cc-kube-api-access-sk4dq\" (UniqueName: \"kubernetes.io/projected/09d9aa32-f70e-4e76-9316-6f58de2a65cc-kube-api-access-sk4dq\") pod \"09d9aa32-f70e-4e76-9316-6f58de2a65cc\" (UID: \"09d9aa32-f70e-4e76-9316-6f58de2a65cc\") "
Oct 30 19:22:40.088372 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:40.088340    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09d9aa32-f70e-4e76-9316-6f58de2a65cc-kube-api-access-sk4dq" pod "09d9aa32-f70e-4e76-9316-6f58de2a65cc" (UID: "09d9aa32-f70e-4e76-9316-6f58de2a65cc"). InnerVolumeSpecName "kube-api-access-sk4dq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:40.187201 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:40.187162    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk4dq\" (UniqueName: \"kubernetes.io/projected/09d9aa32-f70e-4e76-9316-6f58de2a65cc-kube-api-access-sk4dq\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:40.550825 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:40.550792    5043 generic.go:356] "Generic (PLEG): container finished" podID="09d9aa32-f70e-4e76-9316-6f58de2a65cc" containerID="b70841909ccec8da9105fd3f4d6c5598f3f16f3dbc896e05b8688195b556556b" exitCode=0
Oct 30 19:22:40.551235 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:40.550839    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-c74vl"
Oct 30 19:22:40.551235 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:40.550887    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-c74vl" event={"ID":"09d9aa32-f70e-4e76-9316-6f58de2a65cc","Type":"ContainerDied","Data":"b70841909ccec8da9105fd3f4d6c5598f3f16f3dbc896e05b8688195b556556b"}
Oct 30 19:22:40.551235 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:40.550930    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-c74vl" event={"ID":"09d9aa32-f70e-4e76-9316-6f58de2a65cc","Type":"ContainerDied","Data":"ec04c5db2dae560031193fc7b376972ee391095e3b44ba6916c1dd2d46644458"}
Oct 30 19:22:40.551235 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:40.550947    5043 scope.go:122] "RemoveContainer" containerID="b70841909ccec8da9105fd3f4d6c5598f3f16f3dbc896e05b8688195b556556b"
Oct 30 19:22:40.560206 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:40.560192    5043 scope.go:122] "RemoveContainer" containerID="b70841909ccec8da9105fd3f4d6c5598f3f16f3dbc896e05b8688195b556556b"
Oct 30 19:22:40.560458 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:22:40.560439    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b70841909ccec8da9105fd3f4d6c5598f3f16f3dbc896e05b8688195b556556b\": container with ID starting with b70841909ccec8da9105fd3f4d6c5598f3f16f3dbc896e05b8688195b556556b not found: ID does not exist" containerID="b70841909ccec8da9105fd3f4d6c5598f3f16f3dbc896e05b8688195b556556b"
Oct 30 19:22:40.560524 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:40.560466    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b70841909ccec8da9105fd3f4d6c5598f3f16f3dbc896e05b8688195b556556b"} err="failed to get container status \"b70841909ccec8da9105fd3f4d6c5598f3f16f3dbc896e05b8688195b556556b\": rpc error: code = NotFound desc = could not find container \"b70841909ccec8da9105fd3f4d6c5598f3f16f3dbc896e05b8688195b556556b\": container with ID starting with b70841909ccec8da9105fd3f4d6c5598f3f16f3dbc896e05b8688195b556556b not found: ID does not exist"
Oct 30 19:22:40.572898 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:40.572878    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-c74vl"]
Oct 30 19:22:40.577598 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:40.577580    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-c74vl"]
Oct 30 19:22:40.643843 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:22:40.643813    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09d9aa32-f70e-4e76-9316-6f58de2a65cc" path="/var/lib/kubelet/pods/09d9aa32-f70e-4e76-9316-6f58de2a65cc/volumes"
Oct 30 19:22:54.975102 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:22:54.975061    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:23:24.975230 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:23:24.975185    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:23:54.978098 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:23:54.978048    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:23:59.627327 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:23:59.627298    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-9426l"]
Oct 30 19:23:59.627711 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:23:59.627693    5043 cpu_manager.go:414] "containerMap: removing container" podUID="09d9aa32-f70e-4e76-9316-6f58de2a65cc" containerName="authorino"
Oct 30 19:23:59.627711 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:23:59.627705    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09d9aa32-f70e-4e76-9316-6f58de2a65cc" containerName="authorino"
Oct 30 19:23:59.627811 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:23:59.627800    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="09d9aa32-f70e-4e76-9316-6f58de2a65cc" containerName="authorino"
Oct 30 19:23:59.630941 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:23:59.630926    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9426l"
Oct 30 19:23:59.642959 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:23:59.642934    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-w2kzv\""
Oct 30 19:23:59.644970 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:23:59.644942    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Oct 30 19:23:59.644970 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:23:59.644963    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 19:23:59.645170 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:23:59.645157    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 19:23:59.645950 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:23:59.645930    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-9426l"]
Oct 30 19:23:59.649603 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:23:59.649579    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a57ae4e7-3e49-43ab-80a1-6db3d2c058d9-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-9426l\" (UID: \"a57ae4e7-3e49-43ab-80a1-6db3d2c058d9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9426l"
Oct 30 19:23:59.649707 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:23:59.649645    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rsb6\" (UniqueName: \"kubernetes.io/projected/a57ae4e7-3e49-43ab-80a1-6db3d2c058d9-kube-api-access-7rsb6\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-9426l\" (UID: \"a57ae4e7-3e49-43ab-80a1-6db3d2c058d9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9426l"
Oct 30 19:23:59.649707 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:23:59.649674    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a57ae4e7-3e49-43ab-80a1-6db3d2c058d9-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-9426l\" (UID: \"a57ae4e7-3e49-43ab-80a1-6db3d2c058d9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9426l"
Oct 30 19:23:59.649891 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:23:59.649715    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a57ae4e7-3e49-43ab-80a1-6db3d2c058d9-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-9426l\" (UID: \"a57ae4e7-3e49-43ab-80a1-6db3d2c058d9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9426l"
Oct 30 19:23:59.649891 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:23:59.649800    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a57ae4e7-3e49-43ab-80a1-6db3d2c058d9-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-9426l\" (UID: \"a57ae4e7-3e49-43ab-80a1-6db3d2c058d9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9426l"
Oct 30 19:23:59.649891 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:23:59.649829    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a57ae4e7-3e49-43ab-80a1-6db3d2c058d9-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-9426l\" (UID: \"a57ae4e7-3e49-43ab-80a1-6db3d2c058d9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9426l"
Oct 30 19:23:59.750385 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:23:59.750353    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a57ae4e7-3e49-43ab-80a1-6db3d2c058d9-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-9426l\" (UID: \"a57ae4e7-3e49-43ab-80a1-6db3d2c058d9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9426l"
Oct 30 19:23:59.750385 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:23:59.750390    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a57ae4e7-3e49-43ab-80a1-6db3d2c058d9-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-9426l\" (UID: \"a57ae4e7-3e49-43ab-80a1-6db3d2c058d9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9426l"
Oct 30 19:23:59.750626 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:23:59.750428    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a57ae4e7-3e49-43ab-80a1-6db3d2c058d9-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-9426l\" (UID: \"a57ae4e7-3e49-43ab-80a1-6db3d2c058d9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9426l"
Oct 30 19:23:59.750626 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:23:59.750463    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rsb6\" (UniqueName: \"kubernetes.io/projected/a57ae4e7-3e49-43ab-80a1-6db3d2c058d9-kube-api-access-7rsb6\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-9426l\" (UID: \"a57ae4e7-3e49-43ab-80a1-6db3d2c058d9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9426l"
Oct 30 19:23:59.750626 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:23:59.750481    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a57ae4e7-3e49-43ab-80a1-6db3d2c058d9-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-9426l\" (UID: \"a57ae4e7-3e49-43ab-80a1-6db3d2c058d9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9426l"
Oct 30 19:23:59.750626 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:23:59.750509    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a57ae4e7-3e49-43ab-80a1-6db3d2c058d9-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-9426l\" (UID: \"a57ae4e7-3e49-43ab-80a1-6db3d2c058d9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9426l"
Oct 30 19:23:59.750889 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:23:59.750862    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a57ae4e7-3e49-43ab-80a1-6db3d2c058d9-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-9426l\" (UID: \"a57ae4e7-3e49-43ab-80a1-6db3d2c058d9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9426l"
Oct 30 19:23:59.751052 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:23:59.751028    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a57ae4e7-3e49-43ab-80a1-6db3d2c058d9-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-9426l\" (UID: \"a57ae4e7-3e49-43ab-80a1-6db3d2c058d9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9426l"
Oct 30 19:23:59.751242 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:23:59.751158    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a57ae4e7-3e49-43ab-80a1-6db3d2c058d9-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-9426l\" (UID: \"a57ae4e7-3e49-43ab-80a1-6db3d2c058d9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9426l"
Oct 30 19:23:59.753447 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:23:59.753423    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a57ae4e7-3e49-43ab-80a1-6db3d2c058d9-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-9426l\" (UID: \"a57ae4e7-3e49-43ab-80a1-6db3d2c058d9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9426l"
Oct 30 19:23:59.753558 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:23:59.753539    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a57ae4e7-3e49-43ab-80a1-6db3d2c058d9-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-9426l\" (UID: \"a57ae4e7-3e49-43ab-80a1-6db3d2c058d9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9426l"
Oct 30 19:23:59.760312 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:23:59.760287    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rsb6\" (UniqueName: \"kubernetes.io/projected/a57ae4e7-3e49-43ab-80a1-6db3d2c058d9-kube-api-access-7rsb6\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-9426l\" (UID: \"a57ae4e7-3e49-43ab-80a1-6db3d2c058d9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9426l"
Oct 30 19:23:59.943223 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:23:59.943140    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9426l"
Oct 30 19:24:00.219881 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:00.219849    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-9426l"]
Oct 30 19:24:00.223509 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:24:00.223484    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda57ae4e7_3e49_43ab_80a1_6db3d2c058d9.slice/crio-7ae4e035483cfd62b24683553a1bcfd84f194ca0c4dfd19fe463b5d09b5544c7 WatchSource:0}: Error finding container 7ae4e035483cfd62b24683553a1bcfd84f194ca0c4dfd19fe463b5d09b5544c7: Status 404 returned error can't find the container with id 7ae4e035483cfd62b24683553a1bcfd84f194ca0c4dfd19fe463b5d09b5544c7
Oct 30 19:24:00.935319 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:00.935276    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9426l" event={"ID":"a57ae4e7-3e49-43ab-80a1-6db3d2c058d9","Type":"ContainerStarted","Data":"7ae4e035483cfd62b24683553a1bcfd84f194ca0c4dfd19fe463b5d09b5544c7"}
Oct 30 19:24:01.591653 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:01.591618    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-zt5b2"]
Oct 30 19:24:01.597067 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:01.596452    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zt5b2"
Oct 30 19:24:01.607220 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:01.606124    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Oct 30 19:24:01.612422 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:01.610882    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-zt5b2"]
Oct 30 19:24:01.668759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:01.668694    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a348d1da-3b22-4650-84a5-4e2de27d37e3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zt5b2\" (UID: \"a348d1da-3b22-4650-84a5-4e2de27d37e3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zt5b2"
Oct 30 19:24:01.668920 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:01.668840    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a348d1da-3b22-4650-84a5-4e2de27d37e3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zt5b2\" (UID: \"a348d1da-3b22-4650-84a5-4e2de27d37e3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zt5b2"
Oct 30 19:24:01.668973 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:01.668920    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a348d1da-3b22-4650-84a5-4e2de27d37e3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zt5b2\" (UID: \"a348d1da-3b22-4650-84a5-4e2de27d37e3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zt5b2"
Oct 30 19:24:01.669018 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:01.668994    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a348d1da-3b22-4650-84a5-4e2de27d37e3-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zt5b2\" (UID: \"a348d1da-3b22-4650-84a5-4e2de27d37e3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zt5b2"
Oct 30 19:24:01.669067 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:01.669059    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a348d1da-3b22-4650-84a5-4e2de27d37e3-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zt5b2\" (UID: \"a348d1da-3b22-4650-84a5-4e2de27d37e3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zt5b2"
Oct 30 19:24:01.669100 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:01.669079    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sn8m\" (UniqueName: \"kubernetes.io/projected/a348d1da-3b22-4650-84a5-4e2de27d37e3-kube-api-access-6sn8m\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zt5b2\" (UID: \"a348d1da-3b22-4650-84a5-4e2de27d37e3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zt5b2"
Oct 30 19:24:01.769804 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:01.769705    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a348d1da-3b22-4650-84a5-4e2de27d37e3-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zt5b2\" (UID: \"a348d1da-3b22-4650-84a5-4e2de27d37e3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zt5b2"
Oct 30 19:24:01.770781 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:01.770122    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a348d1da-3b22-4650-84a5-4e2de27d37e3-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zt5b2\" (UID: \"a348d1da-3b22-4650-84a5-4e2de27d37e3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zt5b2"
Oct 30 19:24:01.770781 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:01.770360    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sn8m\" (UniqueName: \"kubernetes.io/projected/a348d1da-3b22-4650-84a5-4e2de27d37e3-kube-api-access-6sn8m\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zt5b2\" (UID: \"a348d1da-3b22-4650-84a5-4e2de27d37e3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zt5b2"
Oct 30 19:24:01.770781 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:01.770511    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a348d1da-3b22-4650-84a5-4e2de27d37e3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zt5b2\" (UID: \"a348d1da-3b22-4650-84a5-4e2de27d37e3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zt5b2"
Oct 30 19:24:01.770781 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:01.770579    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a348d1da-3b22-4650-84a5-4e2de27d37e3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zt5b2\" (UID: \"a348d1da-3b22-4650-84a5-4e2de27d37e3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zt5b2"
Oct 30 19:24:01.770781 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:01.770657    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a348d1da-3b22-4650-84a5-4e2de27d37e3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zt5b2\" (UID: \"a348d1da-3b22-4650-84a5-4e2de27d37e3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zt5b2"
Oct 30 19:24:01.770781 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:01.770692    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a348d1da-3b22-4650-84a5-4e2de27d37e3-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zt5b2\" (UID: \"a348d1da-3b22-4650-84a5-4e2de27d37e3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zt5b2"
Oct 30 19:24:01.771199 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:01.771175    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a348d1da-3b22-4650-84a5-4e2de27d37e3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zt5b2\" (UID: \"a348d1da-3b22-4650-84a5-4e2de27d37e3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zt5b2"
Oct 30 19:24:01.772333 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:01.771658    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a348d1da-3b22-4650-84a5-4e2de27d37e3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zt5b2\" (UID: \"a348d1da-3b22-4650-84a5-4e2de27d37e3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zt5b2"
Oct 30 19:24:01.781772 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:01.775048    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a348d1da-3b22-4650-84a5-4e2de27d37e3-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zt5b2\" (UID: \"a348d1da-3b22-4650-84a5-4e2de27d37e3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zt5b2"
Oct 30 19:24:01.785780 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:01.782570    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a348d1da-3b22-4650-84a5-4e2de27d37e3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zt5b2\" (UID: \"a348d1da-3b22-4650-84a5-4e2de27d37e3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zt5b2"
Oct 30 19:24:01.785780 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:01.784600    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sn8m\" (UniqueName: \"kubernetes.io/projected/a348d1da-3b22-4650-84a5-4e2de27d37e3-kube-api-access-6sn8m\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zt5b2\" (UID: \"a348d1da-3b22-4650-84a5-4e2de27d37e3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zt5b2"
Oct 30 19:24:01.920605 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:01.920392    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zt5b2"
Oct 30 19:24:01.946347 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:01.946313    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9426l" event={"ID":"a57ae4e7-3e49-43ab-80a1-6db3d2c058d9","Type":"ContainerStarted","Data":"fd1b0987df19b209f16abe9a4ee404fb3d71f61080b2e2f79a6298c74301136b"}
Oct 30 19:24:01.971759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:01.971666    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9426l" podStartSLOduration=1.519869458 podStartE2EDuration="2.971645103s" totalImagesPullingTime="1.451775645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:01.968348498 +0000 UTC m=+461.479408464" watchObservedRunningTime="2026-10-30 19:24:01.971645103 +0000 UTC m=+461.482705068"
Oct 30 19:24:02.214834 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:02.214803    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-zt5b2"]
Oct 30 19:24:02.215812 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:24:02.215782    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda348d1da_3b22_4650_84a5_4e2de27d37e3.slice/crio-977fac7b7b5cdb5672cab689098c78df367860b560f83f92f2dae1a2bbc56d15 WatchSource:0}: Error finding container 977fac7b7b5cdb5672cab689098c78df367860b560f83f92f2dae1a2bbc56d15: Status 404 returned error can't find the container with id 977fac7b7b5cdb5672cab689098c78df367860b560f83f92f2dae1a2bbc56d15
Oct 30 19:24:02.943527 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:02.943446    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9426l"
Oct 30 19:24:02.953090 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:02.953058    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zt5b2" event={"ID":"a348d1da-3b22-4650-84a5-4e2de27d37e3","Type":"ContainerStarted","Data":"0bdc9ac2287711800ca799f2b9765b40c1508a9a80cb850eee060738e5394b70"}
Oct 30 19:24:02.953523 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:02.953111    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zt5b2" event={"ID":"a348d1da-3b22-4650-84a5-4e2de27d37e3","Type":"ContainerStarted","Data":"977fac7b7b5cdb5672cab689098c78df367860b560f83f92f2dae1a2bbc56d15"}
Oct 30 19:24:02.976858 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:02.976803    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zt5b2" podStartSLOduration=1.6369590710000002 podStartE2EDuration="1.976784163s" totalImagesPullingTime="339.825092ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:02.974917196 +0000 UTC m=+462.485977161" watchObservedRunningTime="2026-10-30 19:24:02.976784163 +0000 UTC m=+462.487844129"
Oct 30 19:24:03.446359 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:03.446330    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-6dg26"]
Oct 30 19:24:03.449972 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:03.449950    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-6dg26"
Oct 30 19:24:03.462627 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:03.462596    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Oct 30 19:24:03.480105 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:03.480074    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-6dg26"]
Oct 30 19:24:03.488455 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:03.488423    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/692c558e-cc23-4533-b6ad-0f982a82c187-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-6dg26\" (UID: \"692c558e-cc23-4533-b6ad-0f982a82c187\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6dg26"
Oct 30 19:24:03.488584 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:03.488492    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/692c558e-cc23-4533-b6ad-0f982a82c187-home\") pod \"echo-simulator-kserve-64b6f8f96c-6dg26\" (UID: \"692c558e-cc23-4533-b6ad-0f982a82c187\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6dg26"
Oct 30 19:24:03.488584 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:03.488539    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/692c558e-cc23-4533-b6ad-0f982a82c187-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-6dg26\" (UID: \"692c558e-cc23-4533-b6ad-0f982a82c187\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6dg26"
Oct 30 19:24:03.488665 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:03.488605    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/692c558e-cc23-4533-b6ad-0f982a82c187-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-6dg26\" (UID: \"692c558e-cc23-4533-b6ad-0f982a82c187\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6dg26"
Oct 30 19:24:03.488665 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:03.488643    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx9g5\" (UniqueName: \"kubernetes.io/projected/692c558e-cc23-4533-b6ad-0f982a82c187-kube-api-access-zx9g5\") pod \"echo-simulator-kserve-64b6f8f96c-6dg26\" (UID: \"692c558e-cc23-4533-b6ad-0f982a82c187\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6dg26"
Oct 30 19:24:03.488789 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:03.488678    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/692c558e-cc23-4533-b6ad-0f982a82c187-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-6dg26\" (UID: \"692c558e-cc23-4533-b6ad-0f982a82c187\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6dg26"
Oct 30 19:24:03.590297 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:03.590259    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/692c558e-cc23-4533-b6ad-0f982a82c187-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-6dg26\" (UID: \"692c558e-cc23-4533-b6ad-0f982a82c187\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6dg26"
Oct 30 19:24:03.590480 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:03.590405    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/692c558e-cc23-4533-b6ad-0f982a82c187-home\") pod \"echo-simulator-kserve-64b6f8f96c-6dg26\" (UID: \"692c558e-cc23-4533-b6ad-0f982a82c187\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6dg26"
Oct 30 19:24:03.590480 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:03.590461    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/692c558e-cc23-4533-b6ad-0f982a82c187-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-6dg26\" (UID: \"692c558e-cc23-4533-b6ad-0f982a82c187\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6dg26"
Oct 30 19:24:03.590583 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:03.590491    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/692c558e-cc23-4533-b6ad-0f982a82c187-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-6dg26\" (UID: \"692c558e-cc23-4533-b6ad-0f982a82c187\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6dg26"
Oct 30 19:24:03.590638 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:03.590598    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx9g5\" (UniqueName: \"kubernetes.io/projected/692c558e-cc23-4533-b6ad-0f982a82c187-kube-api-access-zx9g5\") pod \"echo-simulator-kserve-64b6f8f96c-6dg26\" (UID: \"692c558e-cc23-4533-b6ad-0f982a82c187\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6dg26"
Oct 30 19:24:03.590703 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:03.590646    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/692c558e-cc23-4533-b6ad-0f982a82c187-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-6dg26\" (UID: \"692c558e-cc23-4533-b6ad-0f982a82c187\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6dg26"
Oct 30 19:24:03.590885 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:03.590863    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/692c558e-cc23-4533-b6ad-0f982a82c187-home\") pod \"echo-simulator-kserve-64b6f8f96c-6dg26\" (UID: \"692c558e-cc23-4533-b6ad-0f982a82c187\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6dg26"
Oct 30 19:24:03.591235 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:03.591210    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/692c558e-cc23-4533-b6ad-0f982a82c187-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-6dg26\" (UID: \"692c558e-cc23-4533-b6ad-0f982a82c187\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6dg26"
Oct 30 19:24:03.591460 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:03.591436    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/692c558e-cc23-4533-b6ad-0f982a82c187-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-6dg26\" (UID: \"692c558e-cc23-4533-b6ad-0f982a82c187\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6dg26"
Oct 30 19:24:03.593460 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:03.593439    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/692c558e-cc23-4533-b6ad-0f982a82c187-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-6dg26\" (UID: \"692c558e-cc23-4533-b6ad-0f982a82c187\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6dg26"
Oct 30 19:24:03.593630 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:03.593527    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/692c558e-cc23-4533-b6ad-0f982a82c187-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-6dg26\" (UID: \"692c558e-cc23-4533-b6ad-0f982a82c187\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6dg26"
Oct 30 19:24:03.605753 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:03.605398    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx9g5\" (UniqueName: \"kubernetes.io/projected/692c558e-cc23-4533-b6ad-0f982a82c187-kube-api-access-zx9g5\") pod \"echo-simulator-kserve-64b6f8f96c-6dg26\" (UID: \"692c558e-cc23-4533-b6ad-0f982a82c187\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6dg26"
Oct 30 19:24:03.766950 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:03.766911    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-6dg26"
Oct 30 19:24:03.921592 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:03.921523    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zt5b2"
Oct 30 19:24:04.053048 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:04.053020    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-6dg26"]
Oct 30 19:24:04.054725 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:24:04.054698    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod692c558e_cc23_4533_b6ad_0f982a82c187.slice/crio-4c3c0480477d3ef7dcb9d55699b93740074724377bdffe170e8dd521969ab279 WatchSource:0}: Error finding container 4c3c0480477d3ef7dcb9d55699b93740074724377bdffe170e8dd521969ab279: Status 404 returned error can't find the container with id 4c3c0480477d3ef7dcb9d55699b93740074724377bdffe170e8dd521969ab279
Oct 30 19:24:04.969598 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:04.969558    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-6dg26" event={"ID":"692c558e-cc23-4533-b6ad-0f982a82c187","Type":"ContainerStarted","Data":"4c3c0480477d3ef7dcb9d55699b93740074724377bdffe170e8dd521969ab279"}
Oct 30 19:24:06.982840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:06.982791    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-6dg26" event={"ID":"692c558e-cc23-4533-b6ad-0f982a82c187","Type":"ContainerStarted","Data":"4b7b1034978c4f802086bba00953a21f5fd446f75aff1730b496ca4baeec4a0e"}
Oct 30 19:24:07.018436 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:07.018377    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-6dg26" podStartSLOduration=1.939843816 podStartE2EDuration="4.018362634s" totalImagesPullingTime="2.078518818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:07.016652006 +0000 UTC m=+466.527711996" watchObservedRunningTime="2026-10-30 19:24:07.018362634 +0000 UTC m=+466.529422598"
Oct 30 19:24:07.767067 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:07.767033    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-6dg26"
Oct 30 19:24:09.943561 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:09.943530    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9426l"
Oct 30 19:24:09.956448 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:09.956424    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9426l"
Oct 30 19:24:10.010865 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:10.010842    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-9426l"
Oct 30 19:24:11.921413 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:11.921330    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zt5b2"
Oct 30 19:24:11.933630 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:11.933604    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zt5b2"
Oct 30 19:24:12.019885 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:12.019853    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zt5b2"
Oct 30 19:24:13.767598 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:13.767566    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-6dg26"
Oct 30 19:24:13.780145 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:13.780119    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-6dg26"
Oct 30 19:24:14.029629 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:14.029560    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-6dg26"
Oct 30 19:24:15.973312 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:15.973269    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b566bdfcc-ngn65"]
Oct 30 19:24:15.977076 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:15.977056    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b566bdfcc-ngn65"
Oct 30 19:24:15.998188 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:15.998165    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b566bdfcc-ngn65"]
Oct 30 19:24:16.111274 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:16.111222    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc4tj\" (UniqueName: \"kubernetes.io/projected/13cdaf87-6f93-41a7-aa9e-3adbb59839d6-kube-api-access-dc4tj\") pod \"authorino-b566bdfcc-ngn65\" (UID: \"13cdaf87-6f93-41a7-aa9e-3adbb59839d6\") " pod="kuadrant-system/authorino-b566bdfcc-ngn65"
Oct 30 19:24:16.111452 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:16.111359    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/13cdaf87-6f93-41a7-aa9e-3adbb59839d6-tls-cert\") pod \"authorino-b566bdfcc-ngn65\" (UID: \"13cdaf87-6f93-41a7-aa9e-3adbb59839d6\") " pod="kuadrant-system/authorino-b566bdfcc-ngn65"
Oct 30 19:24:16.212663 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:16.212632    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc4tj\" (UniqueName: \"kubernetes.io/projected/13cdaf87-6f93-41a7-aa9e-3adbb59839d6-kube-api-access-dc4tj\") pod \"authorino-b566bdfcc-ngn65\" (UID: \"13cdaf87-6f93-41a7-aa9e-3adbb59839d6\") " pod="kuadrant-system/authorino-b566bdfcc-ngn65"
Oct 30 19:24:16.212868 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:16.212747    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/13cdaf87-6f93-41a7-aa9e-3adbb59839d6-tls-cert\") pod \"authorino-b566bdfcc-ngn65\" (UID: \"13cdaf87-6f93-41a7-aa9e-3adbb59839d6\") " pod="kuadrant-system/authorino-b566bdfcc-ngn65"
Oct 30 19:24:16.215363 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:16.215339    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/13cdaf87-6f93-41a7-aa9e-3adbb59839d6-tls-cert\") pod \"authorino-b566bdfcc-ngn65\" (UID: \"13cdaf87-6f93-41a7-aa9e-3adbb59839d6\") " pod="kuadrant-system/authorino-b566bdfcc-ngn65"
Oct 30 19:24:16.224344 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:16.224277    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc4tj\" (UniqueName: \"kubernetes.io/projected/13cdaf87-6f93-41a7-aa9e-3adbb59839d6-kube-api-access-dc4tj\") pod \"authorino-b566bdfcc-ngn65\" (UID: \"13cdaf87-6f93-41a7-aa9e-3adbb59839d6\") " pod="kuadrant-system/authorino-b566bdfcc-ngn65"
Oct 30 19:24:16.288778 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:16.288727    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b566bdfcc-ngn65"
Oct 30 19:24:16.560942 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:16.560921    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b566bdfcc-ngn65"]
Oct 30 19:24:16.563775 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:24:16.563727    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13cdaf87_6f93_41a7_aa9e_3adbb59839d6.slice/crio-d4c99032da0612a816f50b23999413f4b2d4e0bd014c61fba2f7ad18cd70cc16 WatchSource:0}: Error finding container d4c99032da0612a816f50b23999413f4b2d4e0bd014c61fba2f7ad18cd70cc16: Status 404 returned error can't find the container with id d4c99032da0612a816f50b23999413f4b2d4e0bd014c61fba2f7ad18cd70cc16
Oct 30 19:24:17.035592 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:17.035556    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b566bdfcc-ngn65" event={"ID":"13cdaf87-6f93-41a7-aa9e-3adbb59839d6","Type":"ContainerStarted","Data":"d4c99032da0612a816f50b23999413f4b2d4e0bd014c61fba2f7ad18cd70cc16"}
Oct 30 19:24:18.049199 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:18.049157    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b566bdfcc-ngn65" event={"ID":"13cdaf87-6f93-41a7-aa9e-3adbb59839d6","Type":"ContainerStarted","Data":"eb769ed1620a2d0edee04a979e6ba83f82006540ec021a2a3ae3ae3598ea6d7c"}
Oct 30 19:24:18.077699 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:18.077637    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-b566bdfcc-ngn65" podStartSLOduration=2.278793966 podStartE2EDuration="3.077618493s" totalImagesPullingTime="798.824527ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:18.073764933 +0000 UTC m=+477.584824898" watchObservedRunningTime="2026-10-30 19:24:18.077618493 +0000 UTC m=+477.588678462"
Oct 30 19:24:18.108885 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:18.108843    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6cbbd8b8bb-czqvm"]
Oct 30 19:24:18.113296 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:18.113101    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6cbbd8b8bb-czqvm" podUID="69ffd927-711e-460a-94d3-b71b1c51bfce" containerName="authorino" containerID="cri-o://823a2431e599f839dd9129ae5596fd52b6a4507f475765f8ceee2ad923bf11f5" gracePeriod=30
Oct 30 19:24:18.393853 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:18.393830    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cbbd8b8bb-czqvm"
Oct 30 19:24:18.562294 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:18.562260    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69ffd927-711e-460a-94d3-b71b1c51bfce-tls-cert\" (UniqueName: \"kubernetes.io/secret/69ffd927-711e-460a-94d3-b71b1c51bfce-tls-cert\") pod \"69ffd927-711e-460a-94d3-b71b1c51bfce\" (UID: \"69ffd927-711e-460a-94d3-b71b1c51bfce\") "
Oct 30 19:24:18.562465 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:18.562384    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69ffd927-711e-460a-94d3-b71b1c51bfce-kube-api-access-64zk9\" (UniqueName: \"kubernetes.io/projected/69ffd927-711e-460a-94d3-b71b1c51bfce-kube-api-access-64zk9\") pod \"69ffd927-711e-460a-94d3-b71b1c51bfce\" (UID: \"69ffd927-711e-460a-94d3-b71b1c51bfce\") "
Oct 30 19:24:18.564631 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:18.564592    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69ffd927-711e-460a-94d3-b71b1c51bfce-kube-api-access-64zk9" pod "69ffd927-711e-460a-94d3-b71b1c51bfce" (UID: "69ffd927-711e-460a-94d3-b71b1c51bfce"). InnerVolumeSpecName "kube-api-access-64zk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:18.575721 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:18.575688    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69ffd927-711e-460a-94d3-b71b1c51bfce-tls-cert" pod "69ffd927-711e-460a-94d3-b71b1c51bfce" (UID: "69ffd927-711e-460a-94d3-b71b1c51bfce"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:24:18.664042 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:18.664009    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64zk9\" (UniqueName: \"kubernetes.io/projected/69ffd927-711e-460a-94d3-b71b1c51bfce-kube-api-access-64zk9\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:18.664042 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:18.664035    5043 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/69ffd927-711e-460a-94d3-b71b1c51bfce-tls-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:19.055004 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:19.054967    5043 generic.go:356] "Generic (PLEG): container finished" podID="69ffd927-711e-460a-94d3-b71b1c51bfce" containerID="823a2431e599f839dd9129ae5596fd52b6a4507f475765f8ceee2ad923bf11f5" exitCode=0
Oct 30 19:24:19.055004 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:19.055003    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cbbd8b8bb-czqvm"
Oct 30 19:24:19.055503 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:19.055047    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cbbd8b8bb-czqvm" event={"ID":"69ffd927-711e-460a-94d3-b71b1c51bfce","Type":"ContainerDied","Data":"823a2431e599f839dd9129ae5596fd52b6a4507f475765f8ceee2ad923bf11f5"}
Oct 30 19:24:19.055503 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:19.055088    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cbbd8b8bb-czqvm" event={"ID":"69ffd927-711e-460a-94d3-b71b1c51bfce","Type":"ContainerDied","Data":"d8865b8d98864f610476202a2b21b26cbca5a2a66f9cf78b7b5abe7c475bfca6"}
Oct 30 19:24:19.055503 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:19.055105    5043 scope.go:122] "RemoveContainer" containerID="823a2431e599f839dd9129ae5596fd52b6a4507f475765f8ceee2ad923bf11f5"
Oct 30 19:24:19.065621 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:19.065603    5043 scope.go:122] "RemoveContainer" containerID="823a2431e599f839dd9129ae5596fd52b6a4507f475765f8ceee2ad923bf11f5"
Oct 30 19:24:19.065961 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:24:19.065943    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"823a2431e599f839dd9129ae5596fd52b6a4507f475765f8ceee2ad923bf11f5\": container with ID starting with 823a2431e599f839dd9129ae5596fd52b6a4507f475765f8ceee2ad923bf11f5 not found: ID does not exist" containerID="823a2431e599f839dd9129ae5596fd52b6a4507f475765f8ceee2ad923bf11f5"
Oct 30 19:24:19.066032 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:19.065973    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"823a2431e599f839dd9129ae5596fd52b6a4507f475765f8ceee2ad923bf11f5"} err="failed to get container status \"823a2431e599f839dd9129ae5596fd52b6a4507f475765f8ceee2ad923bf11f5\": rpc error: code = NotFound desc = could not find container \"823a2431e599f839dd9129ae5596fd52b6a4507f475765f8ceee2ad923bf11f5\": container with ID starting with 823a2431e599f839dd9129ae5596fd52b6a4507f475765f8ceee2ad923bf11f5 not found: ID does not exist"
Oct 30 19:24:19.070848 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:19.070827    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6cbbd8b8bb-czqvm"]
Oct 30 19:24:19.075003 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:19.074982    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6cbbd8b8bb-czqvm"]
Oct 30 19:24:20.644705 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:20.644658    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69ffd927-711e-460a-94d3-b71b1c51bfce" path="/var/lib/kubelet/pods/69ffd927-711e-460a-94d3-b71b1c51bfce/volumes"
Oct 30 19:24:24.985447 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:24:24.985406    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:24:25.200001 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:25.199968    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-pqrwr"]
Oct 30 19:24:25.200380 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:25.200368    5043 cpu_manager.go:414] "containerMap: removing container" podUID="69ffd927-711e-460a-94d3-b71b1c51bfce" containerName="authorino"
Oct 30 19:24:25.200422 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:25.200382    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69ffd927-711e-460a-94d3-b71b1c51bfce" containerName="authorino"
Oct 30 19:24:25.200456 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:25.200442    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="69ffd927-711e-460a-94d3-b71b1c51bfce" containerName="authorino"
Oct 30 19:24:25.204876 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:25.204860    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-pqrwr"
Oct 30 19:24:25.216946 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:25.216921    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-pqrwr"]
Oct 30 19:24:25.256502 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:25.256408    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 30 19:24:25.256650 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:25.256626    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-g9c8m\""
Oct 30 19:24:25.326815 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:25.326775    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g54zc\" (UniqueName: \"kubernetes.io/projected/fa0d2e55-5431-43be-9c88-014880e5f1c0-kube-api-access-g54zc\") pod \"payload-processing-5f78687d5b-pqrwr\" (UID: \"fa0d2e55-5431-43be-9c88-014880e5f1c0\") " pod="openshift-ingress/payload-processing-5f78687d5b-pqrwr"
Oct 30 19:24:25.326970 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:25.326909    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa0d2e55-5431-43be-9c88-014880e5f1c0-plugins-config-volume\") pod \"payload-processing-5f78687d5b-pqrwr\" (UID: \"fa0d2e55-5431-43be-9c88-014880e5f1c0\") " pod="openshift-ingress/payload-processing-5f78687d5b-pqrwr"
Oct 30 19:24:25.427644 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:25.427610    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa0d2e55-5431-43be-9c88-014880e5f1c0-plugins-config-volume\") pod \"payload-processing-5f78687d5b-pqrwr\" (UID: \"fa0d2e55-5431-43be-9c88-014880e5f1c0\") " pod="openshift-ingress/payload-processing-5f78687d5b-pqrwr"
Oct 30 19:24:25.427846 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:25.427668    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g54zc\" (UniqueName: \"kubernetes.io/projected/fa0d2e55-5431-43be-9c88-014880e5f1c0-kube-api-access-g54zc\") pod \"payload-processing-5f78687d5b-pqrwr\" (UID: \"fa0d2e55-5431-43be-9c88-014880e5f1c0\") " pod="openshift-ingress/payload-processing-5f78687d5b-pqrwr"
Oct 30 19:24:25.428342 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:25.428321    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa0d2e55-5431-43be-9c88-014880e5f1c0-plugins-config-volume\") pod \"payload-processing-5f78687d5b-pqrwr\" (UID: \"fa0d2e55-5431-43be-9c88-014880e5f1c0\") " pod="openshift-ingress/payload-processing-5f78687d5b-pqrwr"
Oct 30 19:24:25.436446 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:25.436422    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g54zc\" (UniqueName: \"kubernetes.io/projected/fa0d2e55-5431-43be-9c88-014880e5f1c0-kube-api-access-g54zc\") pod \"payload-processing-5f78687d5b-pqrwr\" (UID: \"fa0d2e55-5431-43be-9c88-014880e5f1c0\") " pod="openshift-ingress/payload-processing-5f78687d5b-pqrwr"
Oct 30 19:24:25.518425 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:25.518341    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-pqrwr"
Oct 30 19:24:25.802822 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:25.802721    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-pqrwr"]
Oct 30 19:24:25.805885 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:24:25.805850    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa0d2e55_5431_43be_9c88_014880e5f1c0.slice/crio-461f7bd289e35b91acbb3aa8ec317a39902f6750c50b4da93d50bebb9e123c7c WatchSource:0}: Error finding container 461f7bd289e35b91acbb3aa8ec317a39902f6750c50b4da93d50bebb9e123c7c: Status 404 returned error can't find the container with id 461f7bd289e35b91acbb3aa8ec317a39902f6750c50b4da93d50bebb9e123c7c
Oct 30 19:24:26.099443 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:26.099357    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-pqrwr" event={"ID":"fa0d2e55-5431-43be-9c88-014880e5f1c0","Type":"ContainerStarted","Data":"461f7bd289e35b91acbb3aa8ec317a39902f6750c50b4da93d50bebb9e123c7c"}
Oct 30 19:24:28.111804 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:28.111699    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-pqrwr" event={"ID":"fa0d2e55-5431-43be-9c88-014880e5f1c0","Type":"ContainerStarted","Data":"e77870d8fa4816ccb307fe80592e0f092aa6fd9a68ab6da162ca653b04342e80"}
Oct 30 19:24:28.112182 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:28.111806    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-pqrwr"
Oct 30 19:24:28.133294 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:28.133239    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-pqrwr" podStartSLOduration=1.097157811 podStartE2EDuration="3.133220998s" totalImagesPullingTime="2.036063187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:28.130189396 +0000 UTC m=+487.641249362" watchObservedRunningTime="2026-10-30 19:24:28.133220998 +0000 UTC m=+487.644280965"
Oct 30 19:24:39.119792 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:24:39.119692    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-pqrwr"
Oct 30 19:24:54.975406 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:24:54.975373    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:25:24.976582 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:25:24.976545    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:25:26.008085 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.008055    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9"]
Oct 30 19:25:26.025686 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.025657    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9"
Oct 30 19:25:26.029340 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.029312    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9"]
Oct 30 19:25:26.037422 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.037398    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-01b24e-openshift-default-dockercfg-tz4v8\""
Oct 30 19:25:26.129310 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.129270    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp"]
Oct 30 19:25:26.133083 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.133063    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp"
Oct 30 19:25:26.143216 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.143191    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-b623ca-openshift-default-dockercfg-9vknm\""
Oct 30 19:25:26.153212 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.153186    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp"]
Oct 30 19:25:26.158781 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.158715    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/224648b8-553f-44ad-9d9a-3784a7bea898-credential-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9\" (UID: \"224648b8-553f-44ad-9d9a-3784a7bea898\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9"
Oct 30 19:25:26.158909 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.158788    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/224648b8-553f-44ad-9d9a-3784a7bea898-istiod-ca-cert\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9\" (UID: \"224648b8-553f-44ad-9d9a-3784a7bea898\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9"
Oct 30 19:25:26.158909 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.158822    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/224648b8-553f-44ad-9d9a-3784a7bea898-istio-token\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9\" (UID: \"224648b8-553f-44ad-9d9a-3784a7bea898\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9"
Oct 30 19:25:26.158909 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.158868    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/224648b8-553f-44ad-9d9a-3784a7bea898-workload-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9\" (UID: \"224648b8-553f-44ad-9d9a-3784a7bea898\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9"
Oct 30 19:25:26.158909 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.158899    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/224648b8-553f-44ad-9d9a-3784a7bea898-istio-data\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9\" (UID: \"224648b8-553f-44ad-9d9a-3784a7bea898\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9"
Oct 30 19:25:26.159117 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.158927    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/224648b8-553f-44ad-9d9a-3784a7bea898-istio-envoy\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9\" (UID: \"224648b8-553f-44ad-9d9a-3784a7bea898\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9"
Oct 30 19:25:26.159117 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.158953    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/224648b8-553f-44ad-9d9a-3784a7bea898-istio-podinfo\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9\" (UID: \"224648b8-553f-44ad-9d9a-3784a7bea898\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9"
Oct 30 19:25:26.159117 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.158979    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqbx6\" (UniqueName: \"kubernetes.io/projected/224648b8-553f-44ad-9d9a-3784a7bea898-kube-api-access-fqbx6\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9\" (UID: \"224648b8-553f-44ad-9d9a-3784a7bea898\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9"
Oct 30 19:25:26.159117 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.159016    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/224648b8-553f-44ad-9d9a-3784a7bea898-workload-certs\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9\" (UID: \"224648b8-553f-44ad-9d9a-3784a7bea898\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9"
Oct 30 19:25:26.240830 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.240797    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr"]
Oct 30 19:25:26.244834 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.244813    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr"
Oct 30 19:25:26.250067 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.250044    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-60577a-openshift-default-dockercfg-86d5l\""
Oct 30 19:25:26.259790 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.259692    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2299b666-7ae6-452c-bde8-203a6de2e0e2-workload-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp\" (UID: \"2299b666-7ae6-452c-bde8-203a6de2e0e2\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp"
Oct 30 19:25:26.259790 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.259764    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/224648b8-553f-44ad-9d9a-3784a7bea898-workload-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9\" (UID: \"224648b8-553f-44ad-9d9a-3784a7bea898\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9"
Oct 30 19:25:26.259966 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.259879    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2299b666-7ae6-452c-bde8-203a6de2e0e2-credential-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp\" (UID: \"2299b666-7ae6-452c-bde8-203a6de2e0e2\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp"
Oct 30 19:25:26.260015 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.259968    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/224648b8-553f-44ad-9d9a-3784a7bea898-istio-data\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9\" (UID: \"224648b8-553f-44ad-9d9a-3784a7bea898\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9"
Oct 30 19:25:26.260066 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.260013    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2299b666-7ae6-452c-bde8-203a6de2e0e2-istiod-ca-cert\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp\" (UID: \"2299b666-7ae6-452c-bde8-203a6de2e0e2\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp"
Oct 30 19:25:26.260066 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.260036    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/224648b8-553f-44ad-9d9a-3784a7bea898-istio-envoy\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9\" (UID: \"224648b8-553f-44ad-9d9a-3784a7bea898\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9"
Oct 30 19:25:26.260066 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.260060    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2299b666-7ae6-452c-bde8-203a6de2e0e2-istio-podinfo\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp\" (UID: \"2299b666-7ae6-452c-bde8-203a6de2e0e2\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp"
Oct 30 19:25:26.260214 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.260156    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/224648b8-553f-44ad-9d9a-3784a7bea898-workload-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9\" (UID: \"224648b8-553f-44ad-9d9a-3784a7bea898\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9"
Oct 30 19:25:26.260266 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.260248    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb8d6\" (UniqueName: \"kubernetes.io/projected/2299b666-7ae6-452c-bde8-203a6de2e0e2-kube-api-access-zb8d6\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp\" (UID: \"2299b666-7ae6-452c-bde8-203a6de2e0e2\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp"
Oct 30 19:25:26.260317 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.260279    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2299b666-7ae6-452c-bde8-203a6de2e0e2-istio-data\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp\" (UID: \"2299b666-7ae6-452c-bde8-203a6de2e0e2\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp"
Oct 30 19:25:26.260370 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.260316    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/224648b8-553f-44ad-9d9a-3784a7bea898-istio-data\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9\" (UID: \"224648b8-553f-44ad-9d9a-3784a7bea898\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9"
Oct 30 19:25:26.260370 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.260322    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/224648b8-553f-44ad-9d9a-3784a7bea898-credential-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9\" (UID: \"224648b8-553f-44ad-9d9a-3784a7bea898\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9"
Oct 30 19:25:26.260370 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.260347    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/224648b8-553f-44ad-9d9a-3784a7bea898-istiod-ca-cert\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9\" (UID: \"224648b8-553f-44ad-9d9a-3784a7bea898\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9"
Oct 30 19:25:26.260510 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.260387    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/224648b8-553f-44ad-9d9a-3784a7bea898-istio-token\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9\" (UID: \"224648b8-553f-44ad-9d9a-3784a7bea898\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9"
Oct 30 19:25:26.260510 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.260414    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/224648b8-553f-44ad-9d9a-3784a7bea898-istio-podinfo\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9\" (UID: \"224648b8-553f-44ad-9d9a-3784a7bea898\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9"
Oct 30 19:25:26.260808 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.260710    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2299b666-7ae6-452c-bde8-203a6de2e0e2-istio-token\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp\" (UID: \"2299b666-7ae6-452c-bde8-203a6de2e0e2\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp"
Oct 30 19:25:26.260808 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.260768    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2299b666-7ae6-452c-bde8-203a6de2e0e2-workload-certs\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp\" (UID: \"2299b666-7ae6-452c-bde8-203a6de2e0e2\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp"
Oct 30 19:25:26.260808 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.260800    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqbx6\" (UniqueName: \"kubernetes.io/projected/224648b8-553f-44ad-9d9a-3784a7bea898-kube-api-access-fqbx6\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9\" (UID: \"224648b8-553f-44ad-9d9a-3784a7bea898\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9"
Oct 30 19:25:26.260960 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.260834    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/224648b8-553f-44ad-9d9a-3784a7bea898-workload-certs\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9\" (UID: \"224648b8-553f-44ad-9d9a-3784a7bea898\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9"
Oct 30 19:25:26.260960 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.260872    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2299b666-7ae6-452c-bde8-203a6de2e0e2-istio-envoy\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp\" (UID: \"2299b666-7ae6-452c-bde8-203a6de2e0e2\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp"
Oct 30 19:25:26.261059 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.261015    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/224648b8-553f-44ad-9d9a-3784a7bea898-istiod-ca-cert\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9\" (UID: \"224648b8-553f-44ad-9d9a-3784a7bea898\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9"
Oct 30 19:25:26.261542 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.261517    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/224648b8-553f-44ad-9d9a-3784a7bea898-credential-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9\" (UID: \"224648b8-553f-44ad-9d9a-3784a7bea898\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9"
Oct 30 19:25:26.261672 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.261574    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/224648b8-553f-44ad-9d9a-3784a7bea898-workload-certs\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9\" (UID: \"224648b8-553f-44ad-9d9a-3784a7bea898\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9"
Oct 30 19:25:26.263463 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.263438    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/224648b8-553f-44ad-9d9a-3784a7bea898-istio-podinfo\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9\" (UID: \"224648b8-553f-44ad-9d9a-3784a7bea898\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9"
Oct 30 19:25:26.263933 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.263917    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/224648b8-553f-44ad-9d9a-3784a7bea898-istio-envoy\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9\" (UID: \"224648b8-553f-44ad-9d9a-3784a7bea898\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9"
Oct 30 19:25:26.269678 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.269652    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr"]
Oct 30 19:25:26.279410 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.279385    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/224648b8-553f-44ad-9d9a-3784a7bea898-istio-token\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9\" (UID: \"224648b8-553f-44ad-9d9a-3784a7bea898\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9"
Oct 30 19:25:26.279796 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.279779    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqbx6\" (UniqueName: \"kubernetes.io/projected/224648b8-553f-44ad-9d9a-3784a7bea898-kube-api-access-fqbx6\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9\" (UID: \"224648b8-553f-44ad-9d9a-3784a7bea898\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9"
Oct 30 19:25:26.343929 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.343883    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9"
Oct 30 19:25:26.361719 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.361690    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6251d2e1-9d28-4bb8-a667-f415d0f79df0-istiod-ca-cert\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr\" (UID: \"6251d2e1-9d28-4bb8-a667-f415d0f79df0\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr"
Oct 30 19:25:26.361842 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.361772    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2299b666-7ae6-452c-bde8-203a6de2e0e2-workload-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp\" (UID: \"2299b666-7ae6-452c-bde8-203a6de2e0e2\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp"
Oct 30 19:25:26.361899 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.361874    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6251d2e1-9d28-4bb8-a667-f415d0f79df0-istio-data\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr\" (UID: \"6251d2e1-9d28-4bb8-a667-f415d0f79df0\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr"
Oct 30 19:25:26.361952 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.361927    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6251d2e1-9d28-4bb8-a667-f415d0f79df0-workload-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr\" (UID: \"6251d2e1-9d28-4bb8-a667-f415d0f79df0\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr"
Oct 30 19:25:26.362020 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.362004    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2299b666-7ae6-452c-bde8-203a6de2e0e2-credential-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp\" (UID: \"2299b666-7ae6-452c-bde8-203a6de2e0e2\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp"
Oct 30 19:25:26.362103 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.362076    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6251d2e1-9d28-4bb8-a667-f415d0f79df0-istio-podinfo\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr\" (UID: \"6251d2e1-9d28-4bb8-a667-f415d0f79df0\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr"
Oct 30 19:25:26.362165 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.362146    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2299b666-7ae6-452c-bde8-203a6de2e0e2-workload-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp\" (UID: \"2299b666-7ae6-452c-bde8-203a6de2e0e2\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp"
Oct 30 19:25:26.362165 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.362158    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6251d2e1-9d28-4bb8-a667-f415d0f79df0-istio-envoy\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr\" (UID: \"6251d2e1-9d28-4bb8-a667-f415d0f79df0\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr"
Oct 30 19:25:26.362266 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.362186    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w4kq\" (UniqueName: \"kubernetes.io/projected/6251d2e1-9d28-4bb8-a667-f415d0f79df0-kube-api-access-2w4kq\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr\" (UID: \"6251d2e1-9d28-4bb8-a667-f415d0f79df0\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr"
Oct 30 19:25:26.362321 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.362267    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6251d2e1-9d28-4bb8-a667-f415d0f79df0-istio-token\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr\" (UID: \"6251d2e1-9d28-4bb8-a667-f415d0f79df0\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr"
Oct 30 19:25:26.362321 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.362316    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2299b666-7ae6-452c-bde8-203a6de2e0e2-istiod-ca-cert\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp\" (UID: \"2299b666-7ae6-452c-bde8-203a6de2e0e2\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp"
Oct 30 19:25:26.362420 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.362341    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2299b666-7ae6-452c-bde8-203a6de2e0e2-istio-podinfo\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp\" (UID: \"2299b666-7ae6-452c-bde8-203a6de2e0e2\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp"
Oct 30 19:25:26.362420 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.362364    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2299b666-7ae6-452c-bde8-203a6de2e0e2-credential-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp\" (UID: \"2299b666-7ae6-452c-bde8-203a6de2e0e2\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp"
Oct 30 19:25:26.362515 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.362425    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb8d6\" (UniqueName: \"kubernetes.io/projected/2299b666-7ae6-452c-bde8-203a6de2e0e2-kube-api-access-zb8d6\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp\" (UID: \"2299b666-7ae6-452c-bde8-203a6de2e0e2\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp"
Oct 30 19:25:26.362515 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.362460    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6251d2e1-9d28-4bb8-a667-f415d0f79df0-credential-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr\" (UID: \"6251d2e1-9d28-4bb8-a667-f415d0f79df0\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr"
Oct 30 19:25:26.363061 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.362987    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2299b666-7ae6-452c-bde8-203a6de2e0e2-istio-data\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp\" (UID: \"2299b666-7ae6-452c-bde8-203a6de2e0e2\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp"
Oct 30 19:25:26.363145 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.363114    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2299b666-7ae6-452c-bde8-203a6de2e0e2-istio-token\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp\" (UID: \"2299b666-7ae6-452c-bde8-203a6de2e0e2\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp"
Oct 30 19:25:26.363198 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.363141    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6251d2e1-9d28-4bb8-a667-f415d0f79df0-workload-certs\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr\" (UID: \"6251d2e1-9d28-4bb8-a667-f415d0f79df0\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr"
Oct 30 19:25:26.363198 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.363174    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2299b666-7ae6-452c-bde8-203a6de2e0e2-workload-certs\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp\" (UID: \"2299b666-7ae6-452c-bde8-203a6de2e0e2\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp"
Oct 30 19:25:26.363318 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.363282    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2299b666-7ae6-452c-bde8-203a6de2e0e2-istio-envoy\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp\" (UID: \"2299b666-7ae6-452c-bde8-203a6de2e0e2\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp"
Oct 30 19:25:26.364274 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.363388    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2299b666-7ae6-452c-bde8-203a6de2e0e2-istiod-ca-cert\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp\" (UID: \"2299b666-7ae6-452c-bde8-203a6de2e0e2\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp"
Oct 30 19:25:26.364274 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.363953    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2299b666-7ae6-452c-bde8-203a6de2e0e2-istio-data\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp\" (UID: \"2299b666-7ae6-452c-bde8-203a6de2e0e2\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp"
Oct 30 19:25:26.364274 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.364222    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2299b666-7ae6-452c-bde8-203a6de2e0e2-workload-certs\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp\" (UID: \"2299b666-7ae6-452c-bde8-203a6de2e0e2\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp"
Oct 30 19:25:26.365712 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.365586    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2299b666-7ae6-452c-bde8-203a6de2e0e2-istio-podinfo\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp\" (UID: \"2299b666-7ae6-452c-bde8-203a6de2e0e2\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp"
Oct 30 19:25:26.367352 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.367330    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2299b666-7ae6-452c-bde8-203a6de2e0e2-istio-envoy\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp\" (UID: \"2299b666-7ae6-452c-bde8-203a6de2e0e2\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp"
Oct 30 19:25:26.382652 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.382622    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2299b666-7ae6-452c-bde8-203a6de2e0e2-istio-token\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp\" (UID: \"2299b666-7ae6-452c-bde8-203a6de2e0e2\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp"
Oct 30 19:25:26.382953 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.382932    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb8d6\" (UniqueName: \"kubernetes.io/projected/2299b666-7ae6-452c-bde8-203a6de2e0e2-kube-api-access-zb8d6\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp\" (UID: \"2299b666-7ae6-452c-bde8-203a6de2e0e2\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp"
Oct 30 19:25:26.446897 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.446870    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp"
Oct 30 19:25:26.453586 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.453051    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f"]
Oct 30 19:25:26.456849 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.456827    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f"
Oct 30 19:25:26.464254 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.464212    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-419d0a-openshift-default-dockercfg-tghgq\""
Oct 30 19:25:26.464890 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.464868    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6251d2e1-9d28-4bb8-a667-f415d0f79df0-workload-certs\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr\" (UID: \"6251d2e1-9d28-4bb8-a667-f415d0f79df0\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr"
Oct 30 19:25:26.467755 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.465432    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6251d2e1-9d28-4bb8-a667-f415d0f79df0-istiod-ca-cert\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr\" (UID: \"6251d2e1-9d28-4bb8-a667-f415d0f79df0\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr"
Oct 30 19:25:26.467755 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.465348    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6251d2e1-9d28-4bb8-a667-f415d0f79df0-workload-certs\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr\" (UID: \"6251d2e1-9d28-4bb8-a667-f415d0f79df0\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr"
Oct 30 19:25:26.467755 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.465490    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6251d2e1-9d28-4bb8-a667-f415d0f79df0-istio-data\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr\" (UID: \"6251d2e1-9d28-4bb8-a667-f415d0f79df0\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr"
Oct 30 19:25:26.467755 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.465538    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6251d2e1-9d28-4bb8-a667-f415d0f79df0-workload-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr\" (UID: \"6251d2e1-9d28-4bb8-a667-f415d0f79df0\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr"
Oct 30 19:25:26.467755 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.465901    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6251d2e1-9d28-4bb8-a667-f415d0f79df0-istio-data\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr\" (UID: \"6251d2e1-9d28-4bb8-a667-f415d0f79df0\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr"
Oct 30 19:25:26.467755 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.465988    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6251d2e1-9d28-4bb8-a667-f415d0f79df0-istio-podinfo\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr\" (UID: \"6251d2e1-9d28-4bb8-a667-f415d0f79df0\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr"
Oct 30 19:25:26.467755 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.466132    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6251d2e1-9d28-4bb8-a667-f415d0f79df0-istiod-ca-cert\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr\" (UID: \"6251d2e1-9d28-4bb8-a667-f415d0f79df0\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr"
Oct 30 19:25:26.467755 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.466354    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6251d2e1-9d28-4bb8-a667-f415d0f79df0-istio-envoy\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr\" (UID: \"6251d2e1-9d28-4bb8-a667-f415d0f79df0\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr"
Oct 30 19:25:26.467755 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.466387    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w4kq\" (UniqueName: \"kubernetes.io/projected/6251d2e1-9d28-4bb8-a667-f415d0f79df0-kube-api-access-2w4kq\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr\" (UID: \"6251d2e1-9d28-4bb8-a667-f415d0f79df0\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr"
Oct 30 19:25:26.467755 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.466409    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6251d2e1-9d28-4bb8-a667-f415d0f79df0-workload-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr\" (UID: \"6251d2e1-9d28-4bb8-a667-f415d0f79df0\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr"
Oct 30 19:25:26.467755 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.466429    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6251d2e1-9d28-4bb8-a667-f415d0f79df0-istio-token\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr\" (UID: \"6251d2e1-9d28-4bb8-a667-f415d0f79df0\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr"
Oct 30 19:25:26.467755 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.466492    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6251d2e1-9d28-4bb8-a667-f415d0f79df0-credential-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr\" (UID: \"6251d2e1-9d28-4bb8-a667-f415d0f79df0\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr"
Oct 30 19:25:26.467755 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.466879    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6251d2e1-9d28-4bb8-a667-f415d0f79df0-credential-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr\" (UID: \"6251d2e1-9d28-4bb8-a667-f415d0f79df0\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr"
Oct 30 19:25:26.471353 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.471325    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6251d2e1-9d28-4bb8-a667-f415d0f79df0-istio-envoy\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr\" (UID: \"6251d2e1-9d28-4bb8-a667-f415d0f79df0\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr"
Oct 30 19:25:26.475109 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.475080    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6251d2e1-9d28-4bb8-a667-f415d0f79df0-istio-podinfo\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr\" (UID: \"6251d2e1-9d28-4bb8-a667-f415d0f79df0\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr"
Oct 30 19:25:26.501129 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.501103    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w4kq\" (UniqueName: \"kubernetes.io/projected/6251d2e1-9d28-4bb8-a667-f415d0f79df0-kube-api-access-2w4kq\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr\" (UID: \"6251d2e1-9d28-4bb8-a667-f415d0f79df0\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr"
Oct 30 19:25:26.510409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.510316    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6251d2e1-9d28-4bb8-a667-f415d0f79df0-istio-token\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr\" (UID: \"6251d2e1-9d28-4bb8-a667-f415d0f79df0\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr"
Oct 30 19:25:26.510544 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.510527    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f"]
Oct 30 19:25:26.559982 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.559952    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr"
Oct 30 19:25:26.567749 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.567706    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc3d11d-936f-41d9-a35f-adf6ab7f7909-credential-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f\" (UID: \"abc3d11d-936f-41d9-a35f-adf6ab7f7909\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f"
Oct 30 19:25:26.567858 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.567810    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc3d11d-936f-41d9-a35f-adf6ab7f7909-workload-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f\" (UID: \"abc3d11d-936f-41d9-a35f-adf6ab7f7909\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f"
Oct 30 19:25:26.567858 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.567847    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abc3d11d-936f-41d9-a35f-adf6ab7f7909-istiod-ca-cert\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f\" (UID: \"abc3d11d-936f-41d9-a35f-adf6ab7f7909\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f"
Oct 30 19:25:26.567981 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.567944    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcs6p\" (UniqueName: \"kubernetes.io/projected/abc3d11d-936f-41d9-a35f-adf6ab7f7909-kube-api-access-kcs6p\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f\" (UID: \"abc3d11d-936f-41d9-a35f-adf6ab7f7909\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f"
Oct 30 19:25:26.568037 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.568018    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abc3d11d-936f-41d9-a35f-adf6ab7f7909-istio-token\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f\" (UID: \"abc3d11d-936f-41d9-a35f-adf6ab7f7909\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f"
Oct 30 19:25:26.568110 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.568092    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abc3d11d-936f-41d9-a35f-adf6ab7f7909-istio-podinfo\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f\" (UID: \"abc3d11d-936f-41d9-a35f-adf6ab7f7909\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f"
Oct 30 19:25:26.568146 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.568130    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abc3d11d-936f-41d9-a35f-adf6ab7f7909-istio-envoy\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f\" (UID: \"abc3d11d-936f-41d9-a35f-adf6ab7f7909\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f"
Oct 30 19:25:26.568183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.568167    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abc3d11d-936f-41d9-a35f-adf6ab7f7909-workload-certs\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f\" (UID: \"abc3d11d-936f-41d9-a35f-adf6ab7f7909\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f"
Oct 30 19:25:26.568276 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.568261    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abc3d11d-936f-41d9-a35f-adf6ab7f7909-istio-data\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f\" (UID: \"abc3d11d-936f-41d9-a35f-adf6ab7f7909\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f"
Oct 30 19:25:26.660306 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.660279    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9"]
Oct 30 19:25:26.662766 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:25:26.662710    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod224648b8_553f_44ad_9d9a_3784a7bea898.slice/crio-02dac12cdbc43ead9ac8efc1a34ab807df13b544beb31d3a04ff3bb0138e90b3 WatchSource:0}: Error finding container 02dac12cdbc43ead9ac8efc1a34ab807df13b544beb31d3a04ff3bb0138e90b3: Status 404 returned error can't find the container with id 02dac12cdbc43ead9ac8efc1a34ab807df13b544beb31d3a04ff3bb0138e90b3
Oct 30 19:25:26.665213 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.665179    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:25:26.665328 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.665241    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:25:26.665328 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.665275    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:25:26.669797 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.669770    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abc3d11d-936f-41d9-a35f-adf6ab7f7909-istio-data\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f\" (UID: \"abc3d11d-936f-41d9-a35f-adf6ab7f7909\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f"
Oct 30 19:25:26.669909 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.669845    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc3d11d-936f-41d9-a35f-adf6ab7f7909-credential-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f\" (UID: \"abc3d11d-936f-41d9-a35f-adf6ab7f7909\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f"
Oct 30 19:25:26.669909 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.669877    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc3d11d-936f-41d9-a35f-adf6ab7f7909-workload-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f\" (UID: \"abc3d11d-936f-41d9-a35f-adf6ab7f7909\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f"
Oct 30 19:25:26.670024 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.669907    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abc3d11d-936f-41d9-a35f-adf6ab7f7909-istiod-ca-cert\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f\" (UID: \"abc3d11d-936f-41d9-a35f-adf6ab7f7909\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f"
Oct 30 19:25:26.670024 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.669943    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcs6p\" (UniqueName: \"kubernetes.io/projected/abc3d11d-936f-41d9-a35f-adf6ab7f7909-kube-api-access-kcs6p\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f\" (UID: \"abc3d11d-936f-41d9-a35f-adf6ab7f7909\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f"
Oct 30 19:25:26.670024 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.669997    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abc3d11d-936f-41d9-a35f-adf6ab7f7909-istio-token\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f\" (UID: \"abc3d11d-936f-41d9-a35f-adf6ab7f7909\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f"
Oct 30 19:25:26.670161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.670027    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abc3d11d-936f-41d9-a35f-adf6ab7f7909-istio-podinfo\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f\" (UID: \"abc3d11d-936f-41d9-a35f-adf6ab7f7909\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f"
Oct 30 19:25:26.670161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.670061    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abc3d11d-936f-41d9-a35f-adf6ab7f7909-istio-envoy\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f\" (UID: \"abc3d11d-936f-41d9-a35f-adf6ab7f7909\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f"
Oct 30 19:25:26.670161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.670110    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abc3d11d-936f-41d9-a35f-adf6ab7f7909-workload-certs\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f\" (UID: \"abc3d11d-936f-41d9-a35f-adf6ab7f7909\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f"
Oct 30 19:25:26.670310 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.670228    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abc3d11d-936f-41d9-a35f-adf6ab7f7909-istio-data\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f\" (UID: \"abc3d11d-936f-41d9-a35f-adf6ab7f7909\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f"
Oct 30 19:25:26.670310 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.670239    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc3d11d-936f-41d9-a35f-adf6ab7f7909-credential-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f\" (UID: \"abc3d11d-936f-41d9-a35f-adf6ab7f7909\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f"
Oct 30 19:25:26.670457 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.670437    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abc3d11d-936f-41d9-a35f-adf6ab7f7909-workload-certs\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f\" (UID: \"abc3d11d-936f-41d9-a35f-adf6ab7f7909\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f"
Oct 30 19:25:26.670859 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.670727    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abc3d11d-936f-41d9-a35f-adf6ab7f7909-istiod-ca-cert\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f\" (UID: \"abc3d11d-936f-41d9-a35f-adf6ab7f7909\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f"
Oct 30 19:25:26.671153 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.671132    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc3d11d-936f-41d9-a35f-adf6ab7f7909-workload-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f\" (UID: \"abc3d11d-936f-41d9-a35f-adf6ab7f7909\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f"
Oct 30 19:25:26.672678 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.672651    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abc3d11d-936f-41d9-a35f-adf6ab7f7909-istio-envoy\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f\" (UID: \"abc3d11d-936f-41d9-a35f-adf6ab7f7909\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f"
Oct 30 19:25:26.673185 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.673165    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abc3d11d-936f-41d9-a35f-adf6ab7f7909-istio-podinfo\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f\" (UID: \"abc3d11d-936f-41d9-a35f-adf6ab7f7909\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f"
Oct 30 19:25:26.682893 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.682870    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcs6p\" (UniqueName: \"kubernetes.io/projected/abc3d11d-936f-41d9-a35f-adf6ab7f7909-kube-api-access-kcs6p\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f\" (UID: \"abc3d11d-936f-41d9-a35f-adf6ab7f7909\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f"
Oct 30 19:25:26.685874 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.685856    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abc3d11d-936f-41d9-a35f-adf6ab7f7909-istio-token\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f\" (UID: \"abc3d11d-936f-41d9-a35f-adf6ab7f7909\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f"
Oct 30 19:25:26.736542 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.736508    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx"]
Oct 30 19:25:26.740713 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.740688    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx"
Oct 30 19:25:26.772500 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.772406    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx"]
Oct 30 19:25:26.774508 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.774462    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp"]
Oct 30 19:25:26.780016 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.779979    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f"
Oct 30 19:25:26.782505 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.781916    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:25:26.782505 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.782035    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:25:26.782505 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.782087    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:25:26.795935 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.795899    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-ad96a4-openshift-default-dockercfg-6zjhk\""
Oct 30 19:25:26.877996 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.876493    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1ca4728-e019-4147-b9a3-d3f662d98e77-workload-certs\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx\" (UID: \"f1ca4728-e019-4147-b9a3-d3f662d98e77\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx"
Oct 30 19:25:26.877996 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.876747    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ca4728-e019-4147-b9a3-d3f662d98e77-workload-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx\" (UID: \"f1ca4728-e019-4147-b9a3-d3f662d98e77\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx"
Oct 30 19:25:26.877996 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.876794    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1ca4728-e019-4147-b9a3-d3f662d98e77-istiod-ca-cert\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx\" (UID: \"f1ca4728-e019-4147-b9a3-d3f662d98e77\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx"
Oct 30 19:25:26.877996 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.876873    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1ca4728-e019-4147-b9a3-d3f662d98e77-istio-token\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx\" (UID: \"f1ca4728-e019-4147-b9a3-d3f662d98e77\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx"
Oct 30 19:25:26.877996 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.876908    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1ca4728-e019-4147-b9a3-d3f662d98e77-istio-envoy\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx\" (UID: \"f1ca4728-e019-4147-b9a3-d3f662d98e77\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx"
Oct 30 19:25:26.877996 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.876945    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1ca4728-e019-4147-b9a3-d3f662d98e77-istio-podinfo\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx\" (UID: \"f1ca4728-e019-4147-b9a3-d3f662d98e77\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx"
Oct 30 19:25:26.877996 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.876976    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ca4728-e019-4147-b9a3-d3f662d98e77-credential-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx\" (UID: \"f1ca4728-e019-4147-b9a3-d3f662d98e77\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx"
Oct 30 19:25:26.877996 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.877064    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1ca4728-e019-4147-b9a3-d3f662d98e77-istio-data\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx\" (UID: \"f1ca4728-e019-4147-b9a3-d3f662d98e77\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx"
Oct 30 19:25:26.877996 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.877097    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92nbc\" (UniqueName: \"kubernetes.io/projected/f1ca4728-e019-4147-b9a3-d3f662d98e77-kube-api-access-92nbc\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx\" (UID: \"f1ca4728-e019-4147-b9a3-d3f662d98e77\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx"
Oct 30 19:25:26.879112 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.879080    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr"]
Oct 30 19:25:26.889804 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:25:26.889715    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6251d2e1_9d28_4bb8_a667_f415d0f79df0.slice/crio-33778dc570b1aac3800b45d995940b6b45aa7cee3a513b0293061d8ace505b75 WatchSource:0}: Error finding container 33778dc570b1aac3800b45d995940b6b45aa7cee3a513b0293061d8ace505b75: Status 404 returned error can't find the container with id 33778dc570b1aac3800b45d995940b6b45aa7cee3a513b0293061d8ace505b75
Oct 30 19:25:26.894498 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.893025    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:25:26.894498 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.893107    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:25:26.894498 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.893150    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:25:26.979254 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.978612    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1ca4728-e019-4147-b9a3-d3f662d98e77-istio-podinfo\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx\" (UID: \"f1ca4728-e019-4147-b9a3-d3f662d98e77\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx"
Oct 30 19:25:26.979254 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.978648    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ca4728-e019-4147-b9a3-d3f662d98e77-credential-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx\" (UID: \"f1ca4728-e019-4147-b9a3-d3f662d98e77\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx"
Oct 30 19:25:26.979254 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.978690    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1ca4728-e019-4147-b9a3-d3f662d98e77-istio-data\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx\" (UID: \"f1ca4728-e019-4147-b9a3-d3f662d98e77\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx"
Oct 30 19:25:26.979254 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.978707    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92nbc\" (UniqueName: \"kubernetes.io/projected/f1ca4728-e019-4147-b9a3-d3f662d98e77-kube-api-access-92nbc\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx\" (UID: \"f1ca4728-e019-4147-b9a3-d3f662d98e77\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx"
Oct 30 19:25:26.979254 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.978753    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1ca4728-e019-4147-b9a3-d3f662d98e77-workload-certs\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx\" (UID: \"f1ca4728-e019-4147-b9a3-d3f662d98e77\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx"
Oct 30 19:25:26.979254 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.978815    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ca4728-e019-4147-b9a3-d3f662d98e77-workload-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx\" (UID: \"f1ca4728-e019-4147-b9a3-d3f662d98e77\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx"
Oct 30 19:25:26.979254 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.978850    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1ca4728-e019-4147-b9a3-d3f662d98e77-istiod-ca-cert\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx\" (UID: \"f1ca4728-e019-4147-b9a3-d3f662d98e77\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx"
Oct 30 19:25:26.979254 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.978889    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1ca4728-e019-4147-b9a3-d3f662d98e77-istio-token\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx\" (UID: \"f1ca4728-e019-4147-b9a3-d3f662d98e77\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx"
Oct 30 19:25:26.979254 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.978921    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1ca4728-e019-4147-b9a3-d3f662d98e77-istio-envoy\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx\" (UID: \"f1ca4728-e019-4147-b9a3-d3f662d98e77\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx"
Oct 30 19:25:26.979254 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.979148    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ca4728-e019-4147-b9a3-d3f662d98e77-credential-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx\" (UID: \"f1ca4728-e019-4147-b9a3-d3f662d98e77\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx"
Oct 30 19:25:26.979898 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.979784    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1ca4728-e019-4147-b9a3-d3f662d98e77-workload-certs\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx\" (UID: \"f1ca4728-e019-4147-b9a3-d3f662d98e77\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx"
Oct 30 19:25:26.980646 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.980025    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1ca4728-e019-4147-b9a3-d3f662d98e77-istio-data\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx\" (UID: \"f1ca4728-e019-4147-b9a3-d3f662d98e77\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx"
Oct 30 19:25:26.980646 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.980351    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ca4728-e019-4147-b9a3-d3f662d98e77-workload-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx\" (UID: \"f1ca4728-e019-4147-b9a3-d3f662d98e77\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx"
Oct 30 19:25:26.980646 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.980600    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1ca4728-e019-4147-b9a3-d3f662d98e77-istiod-ca-cert\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx\" (UID: \"f1ca4728-e019-4147-b9a3-d3f662d98e77\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx"
Oct 30 19:25:26.982184 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.982126    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1ca4728-e019-4147-b9a3-d3f662d98e77-istio-envoy\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx\" (UID: \"f1ca4728-e019-4147-b9a3-d3f662d98e77\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx"
Oct 30 19:25:26.984145 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.984109    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1ca4728-e019-4147-b9a3-d3f662d98e77-istio-podinfo\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx\" (UID: \"f1ca4728-e019-4147-b9a3-d3f662d98e77\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx"
Oct 30 19:25:26.992276 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.992164    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1ca4728-e019-4147-b9a3-d3f662d98e77-istio-token\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx\" (UID: \"f1ca4728-e019-4147-b9a3-d3f662d98e77\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx"
Oct 30 19:25:26.992276 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:26.992241    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92nbc\" (UniqueName: \"kubernetes.io/projected/f1ca4728-e019-4147-b9a3-d3f662d98e77-kube-api-access-92nbc\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx\" (UID: \"f1ca4728-e019-4147-b9a3-d3f662d98e77\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx"
Oct 30 19:25:27.059253 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:27.059180    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx"
Oct 30 19:25:27.142064 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:27.141981    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f"]
Oct 30 19:25:27.153443 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:27.153408    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:25:27.153747 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:27.153713    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:25:27.153875 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:27.153860    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:25:27.404401 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:27.404364    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx"]
Oct 30 19:25:27.406343 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:25:27.406290    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1ca4728_e019_4147_b9a3_d3f662d98e77.slice/crio-5c2965dadde5d1cf97a7999a92b1699c1c664594ce6358ced7b24a11b2a83af4 WatchSource:0}: Error finding container 5c2965dadde5d1cf97a7999a92b1699c1c664594ce6358ced7b24a11b2a83af4: Status 404 returned error can't find the container with id 5c2965dadde5d1cf97a7999a92b1699c1c664594ce6358ced7b24a11b2a83af4
Oct 30 19:25:27.408992 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:27.408959    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:25:27.409084 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:27.409028    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:25:27.409084 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:27.409056    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:25:27.419674 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:27.419640    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx" event={"ID":"f1ca4728-e019-4147-b9a3-d3f662d98e77","Type":"ContainerStarted","Data":"5c2965dadde5d1cf97a7999a92b1699c1c664594ce6358ced7b24a11b2a83af4"}
Oct 30 19:25:27.421496 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:27.421466    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp" event={"ID":"2299b666-7ae6-452c-bde8-203a6de2e0e2","Type":"ContainerStarted","Data":"42320e18a73d22e0f6d2ecea24d2f4c60977246febcac7def4b7fe01f4b077a4"}
Oct 30 19:25:27.421613 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:27.421523    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp" event={"ID":"2299b666-7ae6-452c-bde8-203a6de2e0e2","Type":"ContainerStarted","Data":"3db8aae4b836013528387e54206961a3d6b5ae0d0c505e8058c287fa9404acb8"}
Oct 30 19:25:27.423214 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:27.423187    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f" event={"ID":"abc3d11d-936f-41d9-a35f-adf6ab7f7909","Type":"ContainerStarted","Data":"ec24646caaf543e078ff439bd52db9f9346d5fef8bca7acb0ddfc9faa3f2908e"}
Oct 30 19:25:27.423328 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:27.423237    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f" event={"ID":"abc3d11d-936f-41d9-a35f-adf6ab7f7909","Type":"ContainerStarted","Data":"0a759aaae016799c70b64f9d4209875160c221275ae45d16e760738cddc84b6b"}
Oct 30 19:25:27.425093 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:27.425028    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9" event={"ID":"224648b8-553f-44ad-9d9a-3784a7bea898","Type":"ContainerStarted","Data":"8504037de69583964b9cded037a0e8cca9e67eb62d1504d8e2ad1745a0effd54"}
Oct 30 19:25:27.425297 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:27.425255    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9" event={"ID":"224648b8-553f-44ad-9d9a-3784a7bea898","Type":"ContainerStarted","Data":"02dac12cdbc43ead9ac8efc1a34ab807df13b544beb31d3a04ff3bb0138e90b3"}
Oct 30 19:25:27.428698 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:27.428321    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr" event={"ID":"6251d2e1-9d28-4bb8-a667-f415d0f79df0","Type":"ContainerStarted","Data":"d17141ca8ee5837753e4a10eddd4997368f9905465698c137e35f00095d4e7e3"}
Oct 30 19:25:27.428698 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:27.428365    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr" event={"ID":"6251d2e1-9d28-4bb8-a667-f415d0f79df0","Type":"ContainerStarted","Data":"33778dc570b1aac3800b45d995940b6b45aa7cee3a513b0293061d8ace505b75"}
Oct 30 19:25:27.446653 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:27.446594    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp" podStartSLOduration=1.446573439 podStartE2EDuration="1.446573439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:27.444432691 +0000 UTC m=+546.955492656" watchObservedRunningTime="2026-10-30 19:25:27.446573439 +0000 UTC m=+546.957633404"
Oct 30 19:25:27.447720 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:27.447695    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp"
Oct 30 19:25:27.453875 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:27.453853    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp"
Oct 30 19:25:27.492936 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:27.492889    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f" podStartSLOduration=1.49287454 podStartE2EDuration="1.49287454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:27.468388818 +0000 UTC m=+546.979448786" watchObservedRunningTime="2026-10-30 19:25:27.49287454 +0000 UTC m=+547.003934504"
Oct 30 19:25:27.493287 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:27.493254    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr" podStartSLOduration=1.49324442 podStartE2EDuration="1.49324442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:27.49001979 +0000 UTC m=+547.001079761" watchObservedRunningTime="2026-10-30 19:25:27.49324442 +0000 UTC m=+547.004304384"
Oct 30 19:25:27.517572 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:27.517512    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9" podStartSLOduration=2.517494204 podStartE2EDuration="2.517494204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:27.515415799 +0000 UTC m=+547.026475784" watchObservedRunningTime="2026-10-30 19:25:27.517494204 +0000 UTC m=+547.028554170"
Oct 30 19:25:27.560308 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:27.560271    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr"
Oct 30 19:25:27.568050 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:27.568020    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr"
Oct 30 19:25:27.780692 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:27.780652    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f"
Oct 30 19:25:28.344791 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:28.344754    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9"
Oct 30 19:25:28.350723 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:28.350693    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9"
Oct 30 19:25:28.435189 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:28.435150    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx" event={"ID":"f1ca4728-e019-4147-b9a3-d3f662d98e77","Type":"ContainerStarted","Data":"e5e78c0bd3b1fa3bd878f6ac7958d9146cfea1dbaa9cdc918d7e310f9098d798"}
Oct 30 19:25:28.435430 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:28.435394    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9"
Oct 30 19:25:28.435779 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:28.435759    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr"
Oct 30 19:25:28.438287 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:28.437139    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp"
Oct 30 19:25:28.438287 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:28.437334    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9"
Oct 30 19:25:28.438287 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:28.437543    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr"
Oct 30 19:25:28.438511 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:28.438490    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp"
Oct 30 19:25:28.462624 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:28.462566    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx" podStartSLOduration=2.462553265 podStartE2EDuration="2.462553265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:28.461404416 +0000 UTC m=+547.972464381" watchObservedRunningTime="2026-10-30 19:25:28.462553265 +0000 UTC m=+547.973613229"
Oct 30 19:25:28.786673 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:28.786636    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f"
Oct 30 19:25:29.060094 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:29.060007    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx"
Oct 30 19:25:29.064910 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:29.064883    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx"
Oct 30 19:25:29.443050 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:29.442869    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f"
Oct 30 19:25:29.443050 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:29.442948    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx"
Oct 30 19:25:29.443482 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:29.443239    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx"
Oct 30 19:25:29.443482 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:29.443297    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f"
Oct 30 19:25:35.348239 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:35.348207    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ad96a4-746867c7d7-lnlgf"]
Oct 30 19:25:35.352222 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:35.352201    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ad96a4-746867c7d7-lnlgf"
Oct 30 19:25:35.361650 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:35.361625    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-ad96a4\""
Oct 30 19:25:35.361954 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:35.361923    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-ad96a4-dockercfg-lckc4\""
Oct 30 19:25:35.368337 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:35.368313    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ad96a4-746867c7d7-lnlgf"]
Oct 30 19:25:35.370110 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:35.370084    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bswvb\" (UniqueName: \"kubernetes.io/projected/33ae4933-1096-419e-81c6-defab7a8d6e0-kube-api-access-bswvb\") pod \"payload-pre-processing-e2e-worker-w4-ad96a4-746867c7d7-lnlgf\" (UID: \"33ae4933-1096-419e-81c6-defab7a8d6e0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ad96a4-746867c7d7-lnlgf"
Oct 30 19:25:35.370210 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:35.370133    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33ae4933-1096-419e-81c6-defab7a8d6e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-ad96a4-746867c7d7-lnlgf\" (UID: \"33ae4933-1096-419e-81c6-defab7a8d6e0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ad96a4-746867c7d7-lnlgf"
Oct 30 19:25:35.472013 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:35.471970    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bswvb\" (UniqueName: \"kubernetes.io/projected/33ae4933-1096-419e-81c6-defab7a8d6e0-kube-api-access-bswvb\") pod \"payload-pre-processing-e2e-worker-w4-ad96a4-746867c7d7-lnlgf\" (UID: \"33ae4933-1096-419e-81c6-defab7a8d6e0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ad96a4-746867c7d7-lnlgf"
Oct 30 19:25:35.472223 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:35.472039    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33ae4933-1096-419e-81c6-defab7a8d6e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-ad96a4-746867c7d7-lnlgf\" (UID: \"33ae4933-1096-419e-81c6-defab7a8d6e0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ad96a4-746867c7d7-lnlgf"
Oct 30 19:25:35.473047 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:35.473021    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33ae4933-1096-419e-81c6-defab7a8d6e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-ad96a4-746867c7d7-lnlgf\" (UID: \"33ae4933-1096-419e-81c6-defab7a8d6e0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ad96a4-746867c7d7-lnlgf"
Oct 30 19:25:35.485028 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:35.484991    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bswvb\" (UniqueName: \"kubernetes.io/projected/33ae4933-1096-419e-81c6-defab7a8d6e0-kube-api-access-bswvb\") pod \"payload-pre-processing-e2e-worker-w4-ad96a4-746867c7d7-lnlgf\" (UID: \"33ae4933-1096-419e-81c6-defab7a8d6e0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ad96a4-746867c7d7-lnlgf"
Oct 30 19:25:35.670674 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:35.670555    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ad96a4-746867c7d7-lnlgf"
Oct 30 19:25:35.968390 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:35.968338    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ad96a4-746867c7d7-lnlgf"]
Oct 30 19:25:35.973161 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:25:35.973120    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33ae4933_1096_419e_81c6_defab7a8d6e0.slice/crio-4ce1e5eac8049b0d49a8909a1e88cc0784c49fc44fe92813b1761e9b8e2e6d8d WatchSource:0}: Error finding container 4ce1e5eac8049b0d49a8909a1e88cc0784c49fc44fe92813b1761e9b8e2e6d8d: Status 404 returned error can't find the container with id 4ce1e5eac8049b0d49a8909a1e88cc0784c49fc44fe92813b1761e9b8e2e6d8d
Oct 30 19:25:36.067918 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:36.067895    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ae63bf-56c67d7d7b-mgtt5"]
Oct 30 19:25:36.071534 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:36.071514    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ae63bf-56c67d7d7b-mgtt5"
Oct 30 19:25:36.083055 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:36.080620    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-ae63bf\""
Oct 30 19:25:36.083055 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:36.081474    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ae63bf-56c67d7d7b-mgtt5"]
Oct 30 19:25:36.088558 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:36.088522    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a040422-5cca-40da-87f7-f5451e7e4b5a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-ae63bf-56c67d7d7b-mgtt5\" (UID: \"9a040422-5cca-40da-87f7-f5451e7e4b5a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ae63bf-56c67d7d7b-mgtt5"
Oct 30 19:25:36.088688 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:36.088595    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjn8p\" (UniqueName: \"kubernetes.io/projected/9a040422-5cca-40da-87f7-f5451e7e4b5a-kube-api-access-cjn8p\") pod \"payload-processing-e2e-worker-w0-ae63bf-56c67d7d7b-mgtt5\" (UID: \"9a040422-5cca-40da-87f7-f5451e7e4b5a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ae63bf-56c67d7d7b-mgtt5"
Oct 30 19:25:36.110789 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:36.110753    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ae63bf-76d99cc4d7-fdtz4"]
Oct 30 19:25:36.115290 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:36.115272    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ae63bf-76d99cc4d7-fdtz4"
Oct 30 19:25:36.123822 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:36.123617    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-ae63bf-dockercfg-hgx4d\""
Oct 30 19:25:36.126414 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:36.126391    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ae63bf-76d99cc4d7-fdtz4"]
Oct 30 19:25:36.190048 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:36.190007    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c907135-5b39-45a4-b7b5-c54336e1c1a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ae63bf-76d99cc4d7-fdtz4\" (UID: \"2c907135-5b39-45a4-b7b5-c54336e1c1a1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ae63bf-76d99cc4d7-fdtz4"
Oct 30 19:25:36.190369 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:36.190342    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjn8p\" (UniqueName: \"kubernetes.io/projected/9a040422-5cca-40da-87f7-f5451e7e4b5a-kube-api-access-cjn8p\") pod \"payload-processing-e2e-worker-w0-ae63bf-56c67d7d7b-mgtt5\" (UID: \"9a040422-5cca-40da-87f7-f5451e7e4b5a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ae63bf-56c67d7d7b-mgtt5"
Oct 30 19:25:36.190721 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:36.190691    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hqg9\" (UniqueName: \"kubernetes.io/projected/2c907135-5b39-45a4-b7b5-c54336e1c1a1-kube-api-access-9hqg9\") pod \"payload-pre-processing-e2e-worker-w0-ae63bf-76d99cc4d7-fdtz4\" (UID: \"2c907135-5b39-45a4-b7b5-c54336e1c1a1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ae63bf-76d99cc4d7-fdtz4"
Oct 30 19:25:36.190912 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:36.190894    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a040422-5cca-40da-87f7-f5451e7e4b5a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-ae63bf-56c67d7d7b-mgtt5\" (UID: \"9a040422-5cca-40da-87f7-f5451e7e4b5a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ae63bf-56c67d7d7b-mgtt5"
Oct 30 19:25:36.208234 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:36.208195    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a040422-5cca-40da-87f7-f5451e7e4b5a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-ae63bf-56c67d7d7b-mgtt5\" (UID: \"9a040422-5cca-40da-87f7-f5451e7e4b5a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ae63bf-56c67d7d7b-mgtt5"
Oct 30 19:25:36.220487 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:36.220398    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjn8p\" (UniqueName: \"kubernetes.io/projected/9a040422-5cca-40da-87f7-f5451e7e4b5a-kube-api-access-cjn8p\") pod \"payload-processing-e2e-worker-w0-ae63bf-56c67d7d7b-mgtt5\" (UID: \"9a040422-5cca-40da-87f7-f5451e7e4b5a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ae63bf-56c67d7d7b-mgtt5"
Oct 30 19:25:36.300380 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:36.300339    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c907135-5b39-45a4-b7b5-c54336e1c1a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ae63bf-76d99cc4d7-fdtz4\" (UID: \"2c907135-5b39-45a4-b7b5-c54336e1c1a1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ae63bf-76d99cc4d7-fdtz4"
Oct 30 19:25:36.300780 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:36.300754    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hqg9\" (UniqueName: \"kubernetes.io/projected/2c907135-5b39-45a4-b7b5-c54336e1c1a1-kube-api-access-9hqg9\") pod \"payload-pre-processing-e2e-worker-w0-ae63bf-76d99cc4d7-fdtz4\" (UID: \"2c907135-5b39-45a4-b7b5-c54336e1c1a1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ae63bf-76d99cc4d7-fdtz4"
Oct 30 19:25:36.302096 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:36.302072    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c907135-5b39-45a4-b7b5-c54336e1c1a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ae63bf-76d99cc4d7-fdtz4\" (UID: \"2c907135-5b39-45a4-b7b5-c54336e1c1a1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ae63bf-76d99cc4d7-fdtz4"
Oct 30 19:25:36.310839 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:36.310811    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hqg9\" (UniqueName: \"kubernetes.io/projected/2c907135-5b39-45a4-b7b5-c54336e1c1a1-kube-api-access-9hqg9\") pod \"payload-pre-processing-e2e-worker-w0-ae63bf-76d99cc4d7-fdtz4\" (UID: \"2c907135-5b39-45a4-b7b5-c54336e1c1a1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ae63bf-76d99cc4d7-fdtz4"
Oct 30 19:25:36.386716 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:36.386678    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ae63bf-56c67d7d7b-mgtt5"
Oct 30 19:25:36.429551 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:36.429512    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ae63bf-76d99cc4d7-fdtz4"
Oct 30 19:25:36.489489 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:36.489389    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ad96a4-746867c7d7-lnlgf" event={"ID":"33ae4933-1096-419e-81c6-defab7a8d6e0","Type":"ContainerStarted","Data":"d349cba6d1451d7c8755645219ae5f49b99600ee64526b13e6750669b4fb3ab1"}
Oct 30 19:25:36.489641 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:36.489598    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ad96a4-746867c7d7-lnlgf" event={"ID":"33ae4933-1096-419e-81c6-defab7a8d6e0","Type":"ContainerStarted","Data":"4ce1e5eac8049b0d49a8909a1e88cc0784c49fc44fe92813b1761e9b8e2e6d8d"}
Oct 30 19:25:36.490248 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:36.490224    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ad96a4-746867c7d7-lnlgf"
Oct 30 19:25:36.660856 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:36.660786    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ad96a4-746867c7d7-lnlgf" podStartSLOduration=1.660766723 podStartE2EDuration="1.660766723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:36.513011134 +0000 UTC m=+556.024071111" watchObservedRunningTime="2026-10-30 19:25:36.660766723 +0000 UTC m=+556.171826686"
Oct 30 19:25:36.663999 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:36.663911    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ae63bf-56c67d7d7b-mgtt5"]
Oct 30 19:25:36.666459 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:25:36.666428    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a040422_5cca_40da_87f7_f5451e7e4b5a.slice/crio-9ad651c2f21214b41acea844a22eb39c3262c2f5ab33bdbae7adcefd2268d728 WatchSource:0}: Error finding container 9ad651c2f21214b41acea844a22eb39c3262c2f5ab33bdbae7adcefd2268d728: Status 404 returned error can't find the container with id 9ad651c2f21214b41acea844a22eb39c3262c2f5ab33bdbae7adcefd2268d728
Oct 30 19:25:36.728720 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:36.728693    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ae63bf-76d99cc4d7-fdtz4"]
Oct 30 19:25:36.730560 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:25:36.730536    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c907135_5b39_45a4_b7b5_c54336e1c1a1.slice/crio-b390897aca1b5c28e70e37d1f66672612547358f01a83952299cffaca2665736 WatchSource:0}: Error finding container b390897aca1b5c28e70e37d1f66672612547358f01a83952299cffaca2665736: Status 404 returned error can't find the container with id b390897aca1b5c28e70e37d1f66672612547358f01a83952299cffaca2665736
Oct 30 19:25:37.496277 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:37.496234    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ae63bf-56c67d7d7b-mgtt5" event={"ID":"9a040422-5cca-40da-87f7-f5451e7e4b5a","Type":"ContainerStarted","Data":"f2c8ddca63164304aed8bd4e65f71f4b9689d506d776579460dee070bf0e9306"}
Oct 30 19:25:37.496722 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:37.496427    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-ae63bf-56c67d7d7b-mgtt5"
Oct 30 19:25:37.496722 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:37.496467    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ae63bf-56c67d7d7b-mgtt5" event={"ID":"9a040422-5cca-40da-87f7-f5451e7e4b5a","Type":"ContainerStarted","Data":"9ad651c2f21214b41acea844a22eb39c3262c2f5ab33bdbae7adcefd2268d728"}
Oct 30 19:25:37.497973 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:37.497949    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ae63bf-76d99cc4d7-fdtz4" event={"ID":"2c907135-5b39-45a4-b7b5-c54336e1c1a1","Type":"ContainerStarted","Data":"aaeaa38d81cd644c56300b7336c52716102b1524abfd8701c28bc4ef9ad002d1"}
Oct 30 19:25:37.498075 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:37.497990    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ae63bf-76d99cc4d7-fdtz4" event={"ID":"2c907135-5b39-45a4-b7b5-c54336e1c1a1","Type":"ContainerStarted","Data":"b390897aca1b5c28e70e37d1f66672612547358f01a83952299cffaca2665736"}
Oct 30 19:25:37.498117 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:37.498098    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ae63bf-76d99cc4d7-fdtz4"
Oct 30 19:25:37.515849 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:37.515809    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-ae63bf-56c67d7d7b-mgtt5" podStartSLOduration=1.515797278 podStartE2EDuration="1.515797278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:37.515022042 +0000 UTC m=+557.026082009" watchObservedRunningTime="2026-10-30 19:25:37.515797278 +0000 UTC m=+557.026857243"
Oct 30 19:25:37.531781 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:37.531715    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ae63bf-76d99cc4d7-fdtz4" podStartSLOduration=1.531699248 podStartE2EDuration="1.531699248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:37.531615849 +0000 UTC m=+557.042675815" watchObservedRunningTime="2026-10-30 19:25:37.531699248 +0000 UTC m=+557.042759212"
Oct 30 19:25:48.516791 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:48.515700    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ad96a4-746867c7d7-lnlgf"
Oct 30 19:25:48.516791 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:48.516128    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ae63bf-76d99cc4d7-fdtz4"
Oct 30 19:25:48.517378 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:48.517013    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-ae63bf-56c67d7d7b-mgtt5"
Oct 30 19:25:48.841086 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:48.841002    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"]
Oct 30 19:25:48.851865 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:48.851833    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:25:48.876757 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:48.868340    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-ce69f3\"/\"openshift-service-ca.crt\""
Oct 30 19:25:48.878653 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:48.878618    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"]
Oct 30 19:25:48.913803 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:48.879675    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ce69f3\"/\"simulator-w3-ce69f3-kserve-self-signed-certs\""
Oct 30 19:25:48.913803 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:48.879775    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-ce69f3\"/\"kube-root-ca.crt\""
Oct 30 19:25:48.914258 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:48.914185    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ce69f3\"/\"default-dockercfg-x2w78\""
Oct 30 19:25:48.941246 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:48.941211    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e898751-6f2a-43cf-9d48-f03358f1de63-model-cache\") pod \"simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx\" (UID: \"0e898751-6f2a-43cf-9d48-f03358f1de63\") " pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:25:48.941420 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:48.941261    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e898751-6f2a-43cf-9d48-f03358f1de63-home\") pod \"simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx\" (UID: \"0e898751-6f2a-43cf-9d48-f03358f1de63\") " pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:25:48.941420 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:48.941280    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e898751-6f2a-43cf-9d48-f03358f1de63-dshm\") pod \"simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx\" (UID: \"0e898751-6f2a-43cf-9d48-f03358f1de63\") " pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:25:48.941420 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:48.941359    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e898751-6f2a-43cf-9d48-f03358f1de63-tls-certs\") pod \"simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx\" (UID: \"0e898751-6f2a-43cf-9d48-f03358f1de63\") " pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:25:48.941420 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:48.941381    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sssrl\" (UniqueName: \"kubernetes.io/projected/0e898751-6f2a-43cf-9d48-f03358f1de63-kube-api-access-sssrl\") pod \"simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx\" (UID: \"0e898751-6f2a-43cf-9d48-f03358f1de63\") " pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:25:48.941559 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:48.941428    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e898751-6f2a-43cf-9d48-f03358f1de63-tmp-dir\") pod \"simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx\" (UID: \"0e898751-6f2a-43cf-9d48-f03358f1de63\") " pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:25:49.042108 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:49.042067    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e898751-6f2a-43cf-9d48-f03358f1de63-tmp-dir\") pod \"simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx\" (UID: \"0e898751-6f2a-43cf-9d48-f03358f1de63\") " pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:25:49.042284 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:49.042155    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e898751-6f2a-43cf-9d48-f03358f1de63-model-cache\") pod \"simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx\" (UID: \"0e898751-6f2a-43cf-9d48-f03358f1de63\") " pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:25:49.042284 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:49.042213    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e898751-6f2a-43cf-9d48-f03358f1de63-home\") pod \"simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx\" (UID: \"0e898751-6f2a-43cf-9d48-f03358f1de63\") " pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:25:49.042284 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:49.042243    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e898751-6f2a-43cf-9d48-f03358f1de63-dshm\") pod \"simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx\" (UID: \"0e898751-6f2a-43cf-9d48-f03358f1de63\") " pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:25:49.042406 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:49.042346    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e898751-6f2a-43cf-9d48-f03358f1de63-tls-certs\") pod \"simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx\" (UID: \"0e898751-6f2a-43cf-9d48-f03358f1de63\") " pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:25:49.042406 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:49.042384    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sssrl\" (UniqueName: \"kubernetes.io/projected/0e898751-6f2a-43cf-9d48-f03358f1de63-kube-api-access-sssrl\") pod \"simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx\" (UID: \"0e898751-6f2a-43cf-9d48-f03358f1de63\") " pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:25:49.043512 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:49.043460    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e898751-6f2a-43cf-9d48-f03358f1de63-tmp-dir\") pod \"simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx\" (UID: \"0e898751-6f2a-43cf-9d48-f03358f1de63\") " pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:25:49.043660 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:49.043586    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e898751-6f2a-43cf-9d48-f03358f1de63-home\") pod \"simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx\" (UID: \"0e898751-6f2a-43cf-9d48-f03358f1de63\") " pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:25:49.043995 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:49.043968    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e898751-6f2a-43cf-9d48-f03358f1de63-model-cache\") pod \"simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx\" (UID: \"0e898751-6f2a-43cf-9d48-f03358f1de63\") " pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:25:49.046526 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:49.046502    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e898751-6f2a-43cf-9d48-f03358f1de63-tls-certs\") pod \"simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx\" (UID: \"0e898751-6f2a-43cf-9d48-f03358f1de63\") " pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:25:49.046850 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:49.046833    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e898751-6f2a-43cf-9d48-f03358f1de63-dshm\") pod \"simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx\" (UID: \"0e898751-6f2a-43cf-9d48-f03358f1de63\") " pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:25:49.051434 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:49.051413    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sssrl\" (UniqueName: \"kubernetes.io/projected/0e898751-6f2a-43cf-9d48-f03358f1de63-kube-api-access-sssrl\") pod \"simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx\" (UID: \"0e898751-6f2a-43cf-9d48-f03358f1de63\") " pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:25:49.234260 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:49.233961    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:25:49.622995 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:49.622616    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"]
Oct 30 19:25:49.652764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:49.652714    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" event={"ID":"0e898751-6f2a-43cf-9d48-f03358f1de63","Type":"ContainerStarted","Data":"2ea9cce2661012f9d85e91547bca30862e68243dfd6b7386d9b70e397922edd6"}
Oct 30 19:25:50.684768 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:50.679814    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" event={"ID":"0e898751-6f2a-43cf-9d48-f03358f1de63","Type":"ContainerStarted","Data":"da526e8eeae35cfed2aeeb58503f31186452edf143681be97da5b2f3bfcda7b9"}
Oct 30 19:25:50.733527 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:50.733457    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podStartSLOduration=2.733434365 podStartE2EDuration="2.733434365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:50.732932345 +0000 UTC m=+570.243992304" watchObservedRunningTime="2026-10-30 19:25:50.733434365 +0000 UTC m=+570.244494331"
Oct 30 19:25:51.235640 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:51.235598    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:25:53.728826 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:53.728789    5043 generic.go:356] "Generic (PLEG): container finished" podID="224648b8-553f-44ad-9d9a-3784a7bea898" containerID="8504037de69583964b9cded037a0e8cca9e67eb62d1504d8e2ad1745a0effd54" exitCode=0
Oct 30 19:25:53.729318 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:53.728872    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9" event={"ID":"224648b8-553f-44ad-9d9a-3784a7bea898","Type":"ContainerDied","Data":"8504037de69583964b9cded037a0e8cca9e67eb62d1504d8e2ad1745a0effd54"}
Oct 30 19:25:53.729376 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:53.729360    5043 scope.go:122] "RemoveContainer" containerID="8504037de69583964b9cded037a0e8cca9e67eb62d1504d8e2ad1745a0effd54"
Oct 30 19:25:53.730982 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:53.730947    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:25:53.731064 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:53.731047    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:25:53.731118 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:53.731102    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:25:53.732702 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:53.732681    5043 generic.go:356] "Generic (PLEG): container finished" podID="6251d2e1-9d28-4bb8-a667-f415d0f79df0" containerID="d17141ca8ee5837753e4a10eddd4997368f9905465698c137e35f00095d4e7e3" exitCode=0
Oct 30 19:25:53.732847 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:53.732781    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr" event={"ID":"6251d2e1-9d28-4bb8-a667-f415d0f79df0","Type":"ContainerDied","Data":"d17141ca8ee5837753e4a10eddd4997368f9905465698c137e35f00095d4e7e3"}
Oct 30 19:25:53.733160 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:53.733143    5043 scope.go:122] "RemoveContainer" containerID="d17141ca8ee5837753e4a10eddd4997368f9905465698c137e35f00095d4e7e3"
Oct 30 19:25:53.745846 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:53.745800    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:25:53.745988 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:53.745906    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:25:53.745988 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:53.745959    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:25:54.751234 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:54.751196    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9" event={"ID":"224648b8-553f-44ad-9d9a-3784a7bea898","Type":"ContainerStarted","Data":"4ea546a3bab408e0b58ecbeeab9f86306a7f29763425dcad8cdf95930b5cb1a4"}
Oct 30 19:25:54.754721 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:54.754685    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr" event={"ID":"6251d2e1-9d28-4bb8-a667-f415d0f79df0","Type":"ContainerStarted","Data":"c77d9dd0808935fc5a408dd9c5be241e32f2bdb681f4e19ccf52a5775b214346"}
Oct 30 19:25:55.181819 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:25:55.175364    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:25:55.352431 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:55.351903    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9"
Oct 30 19:25:55.352431 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:55.352074    5043 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Oct 30 19:25:55.352431 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:55.352122    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9" podUID="224648b8-553f-44ad-9d9a-3784a7bea898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Oct 30 19:25:55.353783 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:55.353760    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"]
Oct 30 19:25:55.597761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:55.577996    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"]
Oct 30 19:25:55.597761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:55.578047    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr"
Oct 30 19:25:55.597761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:55.578170    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:25:55.599910 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:55.599162    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-b623ca\"/\"kube-root-ca.crt\""
Oct 30 19:25:55.599910 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:55.599300    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-b623ca\"/\"openshift-service-ca.crt\""
Oct 30 19:25:55.599910 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:55.599435    5043 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Oct 30 19:25:55.599910 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:55.599508    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr" podUID="6251d2e1-9d28-4bb8-a667-f415d0f79df0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Oct 30 19:25:55.599910 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:55.599801    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b623ca\"/\"simulator-w2-b623ca-kserve-self-signed-certs\""
Oct 30 19:25:55.599910 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:55.599859    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b623ca\"/\"default-dockercfg-dlbkj\""
Oct 30 19:25:55.680762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:55.676784    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr4kh\" (UniqueName: \"kubernetes.io/projected/27af3d5f-b68f-4b68-aecf-2fe5f82b4891-kube-api-access-lr4kh\") pod \"simulator-w2-b623ca-kserve-57bf844dd4-d67rs\" (UID: \"27af3d5f-b68f-4b68-aecf-2fe5f82b4891\") " pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:25:55.680762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:55.676839    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27af3d5f-b68f-4b68-aecf-2fe5f82b4891-dshm\") pod \"simulator-w2-b623ca-kserve-57bf844dd4-d67rs\" (UID: \"27af3d5f-b68f-4b68-aecf-2fe5f82b4891\") " pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:25:55.680762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:55.676880    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27af3d5f-b68f-4b68-aecf-2fe5f82b4891-tls-certs\") pod \"simulator-w2-b623ca-kserve-57bf844dd4-d67rs\" (UID: \"27af3d5f-b68f-4b68-aecf-2fe5f82b4891\") " pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:25:55.680762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:55.676919    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27af3d5f-b68f-4b68-aecf-2fe5f82b4891-tmp-dir\") pod \"simulator-w2-b623ca-kserve-57bf844dd4-d67rs\" (UID: \"27af3d5f-b68f-4b68-aecf-2fe5f82b4891\") " pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:25:55.680762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:55.676970    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27af3d5f-b68f-4b68-aecf-2fe5f82b4891-home\") pod \"simulator-w2-b623ca-kserve-57bf844dd4-d67rs\" (UID: \"27af3d5f-b68f-4b68-aecf-2fe5f82b4891\") " pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:25:55.680762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:55.677039    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27af3d5f-b68f-4b68-aecf-2fe5f82b4891-model-cache\") pod \"simulator-w2-b623ca-kserve-57bf844dd4-d67rs\" (UID: \"27af3d5f-b68f-4b68-aecf-2fe5f82b4891\") " pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:25:55.789762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:55.788776    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27af3d5f-b68f-4b68-aecf-2fe5f82b4891-model-cache\") pod \"simulator-w2-b623ca-kserve-57bf844dd4-d67rs\" (UID: \"27af3d5f-b68f-4b68-aecf-2fe5f82b4891\") " pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:25:55.789762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:55.788887    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr4kh\" (UniqueName: \"kubernetes.io/projected/27af3d5f-b68f-4b68-aecf-2fe5f82b4891-kube-api-access-lr4kh\") pod \"simulator-w2-b623ca-kserve-57bf844dd4-d67rs\" (UID: \"27af3d5f-b68f-4b68-aecf-2fe5f82b4891\") " pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:25:55.789762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:55.788922    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27af3d5f-b68f-4b68-aecf-2fe5f82b4891-dshm\") pod \"simulator-w2-b623ca-kserve-57bf844dd4-d67rs\" (UID: \"27af3d5f-b68f-4b68-aecf-2fe5f82b4891\") " pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:25:55.789762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:55.788951    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27af3d5f-b68f-4b68-aecf-2fe5f82b4891-tls-certs\") pod \"simulator-w2-b623ca-kserve-57bf844dd4-d67rs\" (UID: \"27af3d5f-b68f-4b68-aecf-2fe5f82b4891\") " pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:25:55.789762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:55.788986    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27af3d5f-b68f-4b68-aecf-2fe5f82b4891-tmp-dir\") pod \"simulator-w2-b623ca-kserve-57bf844dd4-d67rs\" (UID: \"27af3d5f-b68f-4b68-aecf-2fe5f82b4891\") " pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:25:55.789762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:55.789035    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27af3d5f-b68f-4b68-aecf-2fe5f82b4891-home\") pod \"simulator-w2-b623ca-kserve-57bf844dd4-d67rs\" (UID: \"27af3d5f-b68f-4b68-aecf-2fe5f82b4891\") " pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:25:55.789762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:55.789499    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27af3d5f-b68f-4b68-aecf-2fe5f82b4891-home\") pod \"simulator-w2-b623ca-kserve-57bf844dd4-d67rs\" (UID: \"27af3d5f-b68f-4b68-aecf-2fe5f82b4891\") " pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:25:55.790571 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:55.790276    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27af3d5f-b68f-4b68-aecf-2fe5f82b4891-model-cache\") pod \"simulator-w2-b623ca-kserve-57bf844dd4-d67rs\" (UID: \"27af3d5f-b68f-4b68-aecf-2fe5f82b4891\") " pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:25:55.802815 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:55.792540    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27af3d5f-b68f-4b68-aecf-2fe5f82b4891-tmp-dir\") pod \"simulator-w2-b623ca-kserve-57bf844dd4-d67rs\" (UID: \"27af3d5f-b68f-4b68-aecf-2fe5f82b4891\") " pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:25:55.845767 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:55.836719    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27af3d5f-b68f-4b68-aecf-2fe5f82b4891-dshm\") pod \"simulator-w2-b623ca-kserve-57bf844dd4-d67rs\" (UID: \"27af3d5f-b68f-4b68-aecf-2fe5f82b4891\") " pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:25:55.845767 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:55.838229    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27af3d5f-b68f-4b68-aecf-2fe5f82b4891-tls-certs\") pod \"simulator-w2-b623ca-kserve-57bf844dd4-d67rs\" (UID: \"27af3d5f-b68f-4b68-aecf-2fe5f82b4891\") " pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:25:55.851381 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:55.850564    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr4kh\" (UniqueName: \"kubernetes.io/projected/27af3d5f-b68f-4b68-aecf-2fe5f82b4891-kube-api-access-lr4kh\") pod \"simulator-w2-b623ca-kserve-57bf844dd4-d67rs\" (UID: \"27af3d5f-b68f-4b68-aecf-2fe5f82b4891\") " pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:25:55.980763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:55.979960    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:25:56.350652 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:56.350178    5043 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Oct 30 19:25:56.350652 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:56.350254    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9" podUID="224648b8-553f-44ad-9d9a-3784a7bea898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Oct 30 19:25:56.678760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:56.667392    5043 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Oct 30 19:25:56.678760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:56.667458    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr" podUID="6251d2e1-9d28-4bb8-a667-f415d0f79df0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Oct 30 19:25:56.949077 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:56.948900    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"]
Oct 30 19:25:57.014197 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:25:57.012900    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27af3d5f_b68f_4b68_aecf_2fe5f82b4891.slice/crio-b00e81026f63eaeaa56b453d03700d243e16a1cd718ce521adee5f380cebfa52 WatchSource:0}: Error finding container b00e81026f63eaeaa56b453d03700d243e16a1cd718ce521adee5f380cebfa52: Status 404 returned error can't find the container with id b00e81026f63eaeaa56b453d03700d243e16a1cd718ce521adee5f380cebfa52
Oct 30 19:25:57.345368 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:57.345335    5043 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Oct 30 19:25:57.345474 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:57.345402    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9" podUID="224648b8-553f-44ad-9d9a-3784a7bea898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Oct 30 19:25:57.561779 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:57.560990    5043 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Oct 30 19:25:57.561779 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:57.561048    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr" podUID="6251d2e1-9d28-4bb8-a667-f415d0f79df0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Oct 30 19:25:58.250752 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:58.250691    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"]
Oct 30 19:25:58.351838 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:58.350071    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:25:58.351838 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:58.351792    5043 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Oct 30 19:25:58.352085 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:58.351877    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9" podUID="224648b8-553f-44ad-9d9a-3784a7bea898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Oct 30 19:25:58.365796 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:58.357181    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ae63bf\"/\"kube-root-ca.crt\""
Oct 30 19:25:58.366969 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:58.366448    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ae63bf\"/\"openshift-service-ca.crt\""
Oct 30 19:25:58.366969 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:58.366611    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ae63bf\"/\"default-dockercfg-95md7\""
Oct 30 19:25:58.366969 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:58.366782    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ae63bf\"/\"simulator-w0-ae63bf-kserve-self-signed-certs\""
Oct 30 19:25:58.384871 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:58.381468    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"]
Oct 30 19:25:58.429782 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:58.422749    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96b55519-545c-4336-8bb1-6415d1fdf7f8-tmp-dir\") pod \"simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr\" (UID: \"96b55519-545c-4336-8bb1-6415d1fdf7f8\") " pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:25:58.429782 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:58.422806    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96b55519-545c-4336-8bb1-6415d1fdf7f8-dshm\") pod \"simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr\" (UID: \"96b55519-545c-4336-8bb1-6415d1fdf7f8\") " pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:25:58.429782 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:58.422855    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96b55519-545c-4336-8bb1-6415d1fdf7f8-tls-certs\") pod \"simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr\" (UID: \"96b55519-545c-4336-8bb1-6415d1fdf7f8\") " pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:25:58.429782 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:58.422899    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96b55519-545c-4336-8bb1-6415d1fdf7f8-home\") pod \"simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr\" (UID: \"96b55519-545c-4336-8bb1-6415d1fdf7f8\") " pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:25:58.441462 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:58.422951    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w7b4\" (UniqueName: \"kubernetes.io/projected/96b55519-545c-4336-8bb1-6415d1fdf7f8-kube-api-access-7w7b4\") pod \"simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr\" (UID: \"96b55519-545c-4336-8bb1-6415d1fdf7f8\") " pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:25:58.454761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:58.444649    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96b55519-545c-4336-8bb1-6415d1fdf7f8-model-cache\") pod \"simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr\" (UID: \"96b55519-545c-4336-8bb1-6415d1fdf7f8\") " pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:25:58.483519 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:58.477586    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9"
Oct 30 19:25:58.489809 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:58.488945    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr"
Oct 30 19:25:58.546852 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:58.538106    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" event={"ID":"27af3d5f-b68f-4b68-aecf-2fe5f82b4891","Type":"ContainerStarted","Data":"f2c51ed688b58c5c7e3441d6efd51e87d05bc96c9efe5e573a313573bfda7348"}
Oct 30 19:25:58.546852 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:58.538164    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" event={"ID":"27af3d5f-b68f-4b68-aecf-2fe5f82b4891","Type":"ContainerStarted","Data":"b00e81026f63eaeaa56b453d03700d243e16a1cd718ce521adee5f380cebfa52"}
Oct 30 19:25:58.581762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:58.581651    5043 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Oct 30 19:25:58.597768 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:58.581715    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr" podUID="6251d2e1-9d28-4bb8-a667-f415d0f79df0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Oct 30 19:25:58.597768 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:58.592846    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96b55519-545c-4336-8bb1-6415d1fdf7f8-tmp-dir\") pod \"simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr\" (UID: \"96b55519-545c-4336-8bb1-6415d1fdf7f8\") " pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:25:58.597768 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:58.592901    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96b55519-545c-4336-8bb1-6415d1fdf7f8-dshm\") pod \"simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr\" (UID: \"96b55519-545c-4336-8bb1-6415d1fdf7f8\") " pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:25:58.597768 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:58.592935    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96b55519-545c-4336-8bb1-6415d1fdf7f8-tls-certs\") pod \"simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr\" (UID: \"96b55519-545c-4336-8bb1-6415d1fdf7f8\") " pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:25:58.597768 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:58.592973    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96b55519-545c-4336-8bb1-6415d1fdf7f8-home\") pod \"simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr\" (UID: \"96b55519-545c-4336-8bb1-6415d1fdf7f8\") " pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:25:58.597768 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:58.593028    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w7b4\" (UniqueName: \"kubernetes.io/projected/96b55519-545c-4336-8bb1-6415d1fdf7f8-kube-api-access-7w7b4\") pod \"simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr\" (UID: \"96b55519-545c-4336-8bb1-6415d1fdf7f8\") " pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:25:58.597768 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:58.593208    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96b55519-545c-4336-8bb1-6415d1fdf7f8-model-cache\") pod \"simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr\" (UID: \"96b55519-545c-4336-8bb1-6415d1fdf7f8\") " pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:25:58.599106 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:58.599078    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96b55519-545c-4336-8bb1-6415d1fdf7f8-tmp-dir\") pod \"simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr\" (UID: \"96b55519-545c-4336-8bb1-6415d1fdf7f8\") " pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:25:58.607909 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:58.607878    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96b55519-545c-4336-8bb1-6415d1fdf7f8-home\") pod \"simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr\" (UID: \"96b55519-545c-4336-8bb1-6415d1fdf7f8\") " pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:25:58.629769 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:58.609497    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96b55519-545c-4336-8bb1-6415d1fdf7f8-model-cache\") pod \"simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr\" (UID: \"96b55519-545c-4336-8bb1-6415d1fdf7f8\") " pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:25:58.629769 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:58.628987    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96b55519-545c-4336-8bb1-6415d1fdf7f8-tls-certs\") pod \"simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr\" (UID: \"96b55519-545c-4336-8bb1-6415d1fdf7f8\") " pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:25:58.632492 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:58.632466    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96b55519-545c-4336-8bb1-6415d1fdf7f8-dshm\") pod \"simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr\" (UID: \"96b55519-545c-4336-8bb1-6415d1fdf7f8\") " pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:25:58.673593 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:58.673524    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w7b4\" (UniqueName: \"kubernetes.io/projected/96b55519-545c-4336-8bb1-6415d1fdf7f8-kube-api-access-7w7b4\") pod \"simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr\" (UID: \"96b55519-545c-4336-8bb1-6415d1fdf7f8\") " pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:25:58.846815 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:58.839002    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:25:58.986916 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:58.986830    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:25:59.257766 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:59.238163    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:25:59.376379 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:59.375964    5043 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Oct 30 19:25:59.376379 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:59.376031    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9" podUID="224648b8-553f-44ad-9d9a-3784a7bea898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Oct 30 19:25:59.654237 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:59.653809    5043 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:25:59.793837 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:59.735086    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp" podUID="2299b666-7ae6-452c-bde8-203a6de2e0e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:25:59.793837 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:59.713115    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:25:59.793837 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:59.713519    5043 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Oct 30 19:25:59.793837 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:59.736876    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr" podUID="6251d2e1-9d28-4bb8-a667-f415d0f79df0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Oct 30 19:25:59.858773 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:59.858473    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:25:59.891760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:59.875485    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podStartSLOduration=4.875459193 podStartE2EDuration="4.875459193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:58.581297167 +0000 UTC m=+578.092357114" watchObservedRunningTime="2026-10-30 19:25:59.875459193 +0000 UTC m=+579.386519167"
Oct 30 19:25:59.944690 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:25:59.944605    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"]
Oct 30 19:26:00.345583 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:00.345125    5043 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Oct 30 19:26:00.345583 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:00.345186    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9" podUID="224648b8-553f-44ad-9d9a-3784a7bea898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Oct 30 19:26:00.631333 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:00.630781    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr"
Oct 30 19:26:00.634921 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:00.632862    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr"
Oct 30 19:26:00.651813 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:00.647642    5043 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:26:00.651813 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:00.647711    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx" podUID="f1ca4728-e019-4147-b9a3-d3f662d98e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:26:00.677498 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:00.677457    5043 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:26:00.677686 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:00.677518    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f" podUID="abc3d11d-936f-41d9-a35f-adf6ab7f7909" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:26:00.735987 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:00.735873    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" event={"ID":"96b55519-545c-4336-8bb1-6415d1fdf7f8","Type":"ContainerStarted","Data":"bc08db6b06d31d1745de943fee4254d60cd224ef621a759c01744954b6518bb5"}
Oct 30 19:26:00.735987 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:00.735919    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" event={"ID":"96b55519-545c-4336-8bb1-6415d1fdf7f8","Type":"ContainerStarted","Data":"558d4149c70ff8bec4fe58f7c7e3e32f145cc11b612416bcc009e72410320f17"}
Oct 30 19:26:00.763245 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:00.763177    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podStartSLOduration=2.763156361 podStartE2EDuration="2.763156361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:00.760603587 +0000 UTC m=+580.271663553" watchObservedRunningTime="2026-10-30 19:26:00.763156361 +0000 UTC m=+580.274216327"
Oct 30 19:26:00.841212 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:00.841169    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:26:01.350829 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:01.350795    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9"
Oct 30 19:26:01.352197 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:01.352169    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9"
Oct 30 19:26:02.560769 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:02.560691    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"]
Oct 30 19:26:02.565329 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:02.565302    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:26:02.573197 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:02.573171    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-01b24e\"/\"openshift-service-ca.crt\""
Oct 30 19:26:02.573197 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:02.573171    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-01b24e\"/\"kube-root-ca.crt\""
Oct 30 19:26:02.574209 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:02.574187    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-01b24e\"/\"simulator-w6-01b24e-kserve-self-signed-certs\""
Oct 30 19:26:02.574331 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:02.574207    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-01b24e\"/\"default-dockercfg-tdws5\""
Oct 30 19:26:02.581027 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:02.580998    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"]
Oct 30 19:26:02.758320 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:02.758279    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e62bd6c5-e6ef-427d-a5de-4e2349026a2b-tls-certs\") pod \"simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5\" (UID: \"e62bd6c5-e6ef-427d-a5de-4e2349026a2b\") " pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:26:02.758520 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:02.758360    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e62bd6c5-e6ef-427d-a5de-4e2349026a2b-model-cache\") pod \"simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5\" (UID: \"e62bd6c5-e6ef-427d-a5de-4e2349026a2b\") " pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:26:02.758592 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:02.758561    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e62bd6c5-e6ef-427d-a5de-4e2349026a2b-dshm\") pod \"simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5\" (UID: \"e62bd6c5-e6ef-427d-a5de-4e2349026a2b\") " pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:26:02.758649 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:02.758591    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e62bd6c5-e6ef-427d-a5de-4e2349026a2b-home\") pod \"simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5\" (UID: \"e62bd6c5-e6ef-427d-a5de-4e2349026a2b\") " pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:26:02.758703 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:02.758643    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68pkt\" (UniqueName: \"kubernetes.io/projected/e62bd6c5-e6ef-427d-a5de-4e2349026a2b-kube-api-access-68pkt\") pod \"simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5\" (UID: \"e62bd6c5-e6ef-427d-a5de-4e2349026a2b\") " pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:26:02.758703 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:02.758673    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e62bd6c5-e6ef-427d-a5de-4e2349026a2b-tmp-dir\") pod \"simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5\" (UID: \"e62bd6c5-e6ef-427d-a5de-4e2349026a2b\") " pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:26:02.763518 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:02.763484    5043 generic.go:356] "Generic (PLEG): container finished" podID="f1ca4728-e019-4147-b9a3-d3f662d98e77" containerID="e5e78c0bd3b1fa3bd878f6ac7958d9146cfea1dbaa9cdc918d7e310f9098d798" exitCode=0
Oct 30 19:26:02.763688 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:02.763566    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx" event={"ID":"f1ca4728-e019-4147-b9a3-d3f662d98e77","Type":"ContainerDied","Data":"e5e78c0bd3b1fa3bd878f6ac7958d9146cfea1dbaa9cdc918d7e310f9098d798"}
Oct 30 19:26:02.764094 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:02.764068    5043 scope.go:122] "RemoveContainer" containerID="e5e78c0bd3b1fa3bd878f6ac7958d9146cfea1dbaa9cdc918d7e310f9098d798"
Oct 30 19:26:02.765640 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:02.765605    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:26:02.765757 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:02.765692    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:26:02.765826 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:02.765756    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:26:02.766133 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:02.766103    5043 generic.go:356] "Generic (PLEG): container finished" podID="2299b666-7ae6-452c-bde8-203a6de2e0e2" containerID="42320e18a73d22e0f6d2ecea24d2f4c60977246febcac7def4b7fe01f4b077a4" exitCode=0
Oct 30 19:26:02.766544 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:02.766158    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp" event={"ID":"2299b666-7ae6-452c-bde8-203a6de2e0e2","Type":"ContainerDied","Data":"42320e18a73d22e0f6d2ecea24d2f4c60977246febcac7def4b7fe01f4b077a4"}
Oct 30 19:26:02.767158 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:02.767139    5043 scope.go:122] "RemoveContainer" containerID="42320e18a73d22e0f6d2ecea24d2f4c60977246febcac7def4b7fe01f4b077a4"
Oct 30 19:26:02.769455 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:02.769423    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:26:02.769545 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:02.769531    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:26:02.769641 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:02.769610    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:26:02.859602 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:02.859565    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e62bd6c5-e6ef-427d-a5de-4e2349026a2b-dshm\") pod \"simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5\" (UID: \"e62bd6c5-e6ef-427d-a5de-4e2349026a2b\") " pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:26:02.859856 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:02.859635    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e62bd6c5-e6ef-427d-a5de-4e2349026a2b-home\") pod \"simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5\" (UID: \"e62bd6c5-e6ef-427d-a5de-4e2349026a2b\") " pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:26:02.859856 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:02.859673    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68pkt\" (UniqueName: \"kubernetes.io/projected/e62bd6c5-e6ef-427d-a5de-4e2349026a2b-kube-api-access-68pkt\") pod \"simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5\" (UID: \"e62bd6c5-e6ef-427d-a5de-4e2349026a2b\") " pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:26:02.859856 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:02.859720    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e62bd6c5-e6ef-427d-a5de-4e2349026a2b-tmp-dir\") pod \"simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5\" (UID: \"e62bd6c5-e6ef-427d-a5de-4e2349026a2b\") " pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:26:02.859856 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:02.859811    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e62bd6c5-e6ef-427d-a5de-4e2349026a2b-tls-certs\") pod \"simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5\" (UID: \"e62bd6c5-e6ef-427d-a5de-4e2349026a2b\") " pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:26:02.862631 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:02.860333    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e62bd6c5-e6ef-427d-a5de-4e2349026a2b-tmp-dir\") pod \"simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5\" (UID: \"e62bd6c5-e6ef-427d-a5de-4e2349026a2b\") " pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:26:02.862631 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:02.860537    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e62bd6c5-e6ef-427d-a5de-4e2349026a2b-home\") pod \"simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5\" (UID: \"e62bd6c5-e6ef-427d-a5de-4e2349026a2b\") " pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:26:02.862631 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:02.861237    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e62bd6c5-e6ef-427d-a5de-4e2349026a2b-model-cache\") pod \"simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5\" (UID: \"e62bd6c5-e6ef-427d-a5de-4e2349026a2b\") " pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:26:02.862631 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:02.861548    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e62bd6c5-e6ef-427d-a5de-4e2349026a2b-model-cache\") pod \"simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5\" (UID: \"e62bd6c5-e6ef-427d-a5de-4e2349026a2b\") " pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:26:02.868773 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:02.868721    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e62bd6c5-e6ef-427d-a5de-4e2349026a2b-tls-certs\") pod \"simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5\" (UID: \"e62bd6c5-e6ef-427d-a5de-4e2349026a2b\") " pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:26:02.869167 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:02.869138    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e62bd6c5-e6ef-427d-a5de-4e2349026a2b-dshm\") pod \"simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5\" (UID: \"e62bd6c5-e6ef-427d-a5de-4e2349026a2b\") " pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:26:02.871656 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:02.871633    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68pkt\" (UniqueName: \"kubernetes.io/projected/e62bd6c5-e6ef-427d-a5de-4e2349026a2b-kube-api-access-68pkt\") pod \"simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5\" (UID: \"e62bd6c5-e6ef-427d-a5de-4e2349026a2b\") " pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:26:02.882724 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:02.882651    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:26:03.200024 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:03.199992    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"]
Oct 30 19:26:03.202494 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:26:03.202439    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode62bd6c5_e6ef_427d_a5de_4e2349026a2b.slice/crio-7f1d30b01283a588c0906052f7d4699c39346cd6495cb7512024a9ddeda45fa0 WatchSource:0}: Error finding container 7f1d30b01283a588c0906052f7d4699c39346cd6495cb7512024a9ddeda45fa0: Status 404 returned error can't find the container with id 7f1d30b01283a588c0906052f7d4699c39346cd6495cb7512024a9ddeda45fa0
Oct 30 19:26:03.804201 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:03.804164    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx" event={"ID":"f1ca4728-e019-4147-b9a3-d3f662d98e77","Type":"ContainerStarted","Data":"6927cc37fe5ae78876e2ef49ba2720ba7947b71ead78bcdc4bb804a20073f013"}
Oct 30 19:26:03.808932 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:03.808883    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp" event={"ID":"2299b666-7ae6-452c-bde8-203a6de2e0e2","Type":"ContainerStarted","Data":"34617ed9fdfeed4050ca97494268475c4287df5faa072b235b44741cccd6871b"}
Oct 30 19:26:03.820148 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:03.820121    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" event={"ID":"e62bd6c5-e6ef-427d-a5de-4e2349026a2b","Type":"ContainerStarted","Data":"5a146da3d9f677e3ecaeeb3ee4266570c623883eba84bc3ae2904db3849e85ba"}
Oct 30 19:26:03.820888 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:03.820821    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" event={"ID":"e62bd6c5-e6ef-427d-a5de-4e2349026a2b","Type":"ContainerStarted","Data":"7f1d30b01283a588c0906052f7d4699c39346cd6495cb7512024a9ddeda45fa0"}
Oct 30 19:26:03.884897 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:03.883824    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:26:03.910780 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:03.910668    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podStartSLOduration=1.910635726 podStartE2EDuration="1.910635726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:03.860987697 +0000 UTC m=+583.372047663" watchObservedRunningTime="2026-10-30 19:26:03.910635726 +0000 UTC m=+583.421695696"
Oct 30 19:26:04.064204 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:04.064114    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx"
Oct 30 19:26:04.064590 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:04.064464    5043 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 19:26:04.064792 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:04.064763    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx" podUID="f1ca4728-e019-4147-b9a3-d3f662d98e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 19:26:04.447346 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:04.447254    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp"
Oct 30 19:26:04.447610 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:04.447581    5043 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Oct 30 19:26:04.447695 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:04.447662    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp" podUID="2299b666-7ae6-452c-bde8-203a6de2e0e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Oct 30 19:26:05.070210 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:05.070171    5043 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 19:26:05.081995 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:05.070806    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx" podUID="f1ca4728-e019-4147-b9a3-d3f662d98e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 19:26:05.448055 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:05.447967    5043 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Oct 30 19:26:05.448055 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:05.448022    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp" podUID="2299b666-7ae6-452c-bde8-203a6de2e0e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Oct 30 19:26:05.834347 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:05.834317    5043 generic.go:356] "Generic (PLEG): container finished" podID="abc3d11d-936f-41d9-a35f-adf6ab7f7909" containerID="ec24646caaf543e078ff439bd52db9f9346d5fef8bca7acb0ddfc9faa3f2908e" exitCode=0
Oct 30 19:26:05.834531 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:05.834390    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f" event={"ID":"abc3d11d-936f-41d9-a35f-adf6ab7f7909","Type":"ContainerDied","Data":"ec24646caaf543e078ff439bd52db9f9346d5fef8bca7acb0ddfc9faa3f2908e"}
Oct 30 19:26:05.834979 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:05.834964    5043 scope.go:122] "RemoveContainer" containerID="ec24646caaf543e078ff439bd52db9f9346d5fef8bca7acb0ddfc9faa3f2908e"
Oct 30 19:26:05.836313 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:05.836288    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:26:05.836375 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:05.836363    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:26:05.836412 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:05.836398    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:26:05.990788 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:05.984447    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:26:06.039214 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:06.039070    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:26:06.093817 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:06.092872    5043 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 19:26:06.093817 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:06.092936    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx" podUID="f1ca4728-e019-4147-b9a3-d3f662d98e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 19:26:06.447393 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:06.447304    5043 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Oct 30 19:26:06.447393 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:06.447365    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp" podUID="2299b666-7ae6-452c-bde8-203a6de2e0e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Oct 30 19:26:06.855926 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:06.855783    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f" event={"ID":"abc3d11d-936f-41d9-a35f-adf6ab7f7909","Type":"ContainerStarted","Data":"5df6a3c856c9ec8f23289573e39bb1dbae89bebdf070aa296599acc7e1a65525"}
Oct 30 19:26:06.951221 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:06.951186    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:26:07.072764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:07.070900    5043 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 19:26:07.079769 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:07.074861    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx" podUID="f1ca4728-e019-4147-b9a3-d3f662d98e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 19:26:07.448198 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:07.448160    5043 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Oct 30 19:26:07.448846 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:07.448813    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp" podUID="2299b666-7ae6-452c-bde8-203a6de2e0e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Oct 30 19:26:07.794761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:07.783187    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f"
Oct 30 19:26:07.794761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:07.783358    5043 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 30 19:26:07.794761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:07.783406    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f" podUID="abc3d11d-936f-41d9-a35f-adf6ab7f7909" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 30 19:26:08.075447 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:08.075345    5043 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 19:26:08.075725 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:08.075676    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx" podUID="f1ca4728-e019-4147-b9a3-d3f662d98e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 19:26:08.458826 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:08.450143    5043 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Oct 30 19:26:08.458826 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:08.450220    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp" podUID="2299b666-7ae6-452c-bde8-203a6de2e0e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Oct 30 19:26:08.792763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:08.786144    5043 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 30 19:26:08.792763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:08.786210    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f" podUID="abc3d11d-936f-41d9-a35f-adf6ab7f7909" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 30 19:26:08.846053 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:08.846010    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:26:09.059183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:09.059092    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:26:09.061531 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:09.061501    5043 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 19:26:09.061808 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:09.061725    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx" podUID="f1ca4728-e019-4147-b9a3-d3f662d98e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 19:26:09.081510 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:09.081439    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:26:09.450820 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:09.450160    5043 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Oct 30 19:26:09.450820 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:09.450226    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp" podUID="2299b666-7ae6-452c-bde8-203a6de2e0e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Oct 30 19:26:09.783145 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:09.783102    5043 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 30 19:26:09.783762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:09.783709    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f" podUID="abc3d11d-936f-41d9-a35f-adf6ab7f7909" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 30 19:26:10.316203 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:10.316173    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx"
Oct 30 19:26:10.316814 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:10.316790    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx"
Oct 30 19:26:10.318978 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:10.318960    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx"
Oct 30 19:26:10.453071 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:10.453034    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp"
Oct 30 19:26:10.453492 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:10.453466    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp"
Oct 30 19:26:10.454146 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:10.454127    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp"
Oct 30 19:26:10.788775 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:10.785120    5043 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 30 19:26:10.788775 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:10.785183    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f" podUID="abc3d11d-936f-41d9-a35f-adf6ab7f7909" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 30 19:26:11.786474 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:11.786378    5043 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 30 19:26:11.786474 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:11.786445    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f" podUID="abc3d11d-936f-41d9-a35f-adf6ab7f7909" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 30 19:26:12.226007 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:12.225974    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"]
Oct 30 19:26:12.230043 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:12.230019    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:26:12.244572 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:12.244549    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ce69f3\"/\"premium-w3-ce69f3-kserve-self-signed-certs\""
Oct 30 19:26:12.294071 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:12.294030    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"]
Oct 30 19:26:12.340680 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:12.340639    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/583dd7dc-b160-468a-8ebe-781bead5e989-tmp-dir\") pod \"premium-w3-ce69f3-kserve-7bd85bd65-5z7vf\" (UID: \"583dd7dc-b160-468a-8ebe-781bead5e989\") " pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:26:12.340978 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:12.340950    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/583dd7dc-b160-468a-8ebe-781bead5e989-tls-certs\") pod \"premium-w3-ce69f3-kserve-7bd85bd65-5z7vf\" (UID: \"583dd7dc-b160-468a-8ebe-781bead5e989\") " pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:26:12.341089 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:12.340998    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/583dd7dc-b160-468a-8ebe-781bead5e989-model-cache\") pod \"premium-w3-ce69f3-kserve-7bd85bd65-5z7vf\" (UID: \"583dd7dc-b160-468a-8ebe-781bead5e989\") " pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:26:12.341089 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:12.341074    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lsp9\" (UniqueName: \"kubernetes.io/projected/583dd7dc-b160-468a-8ebe-781bead5e989-kube-api-access-5lsp9\") pod \"premium-w3-ce69f3-kserve-7bd85bd65-5z7vf\" (UID: \"583dd7dc-b160-468a-8ebe-781bead5e989\") " pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:26:12.341168 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:12.341097    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/583dd7dc-b160-468a-8ebe-781bead5e989-home\") pod \"premium-w3-ce69f3-kserve-7bd85bd65-5z7vf\" (UID: \"583dd7dc-b160-468a-8ebe-781bead5e989\") " pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:26:12.341251 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:12.341222    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/583dd7dc-b160-468a-8ebe-781bead5e989-dshm\") pod \"premium-w3-ce69f3-kserve-7bd85bd65-5z7vf\" (UID: \"583dd7dc-b160-468a-8ebe-781bead5e989\") " pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:26:12.442240 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:12.442204    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/583dd7dc-b160-468a-8ebe-781bead5e989-tmp-dir\") pod \"premium-w3-ce69f3-kserve-7bd85bd65-5z7vf\" (UID: \"583dd7dc-b160-468a-8ebe-781bead5e989\") " pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:26:12.442435 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:12.442263    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/583dd7dc-b160-468a-8ebe-781bead5e989-tls-certs\") pod \"premium-w3-ce69f3-kserve-7bd85bd65-5z7vf\" (UID: \"583dd7dc-b160-468a-8ebe-781bead5e989\") " pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:26:12.442435 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:12.442312    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/583dd7dc-b160-468a-8ebe-781bead5e989-model-cache\") pod \"premium-w3-ce69f3-kserve-7bd85bd65-5z7vf\" (UID: \"583dd7dc-b160-468a-8ebe-781bead5e989\") " pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:26:12.442435 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:12.442360    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lsp9\" (UniqueName: \"kubernetes.io/projected/583dd7dc-b160-468a-8ebe-781bead5e989-kube-api-access-5lsp9\") pod \"premium-w3-ce69f3-kserve-7bd85bd65-5z7vf\" (UID: \"583dd7dc-b160-468a-8ebe-781bead5e989\") " pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:26:12.442435 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:12.442382    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/583dd7dc-b160-468a-8ebe-781bead5e989-home\") pod \"premium-w3-ce69f3-kserve-7bd85bd65-5z7vf\" (UID: \"583dd7dc-b160-468a-8ebe-781bead5e989\") " pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:26:12.442435 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:12.442430    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/583dd7dc-b160-468a-8ebe-781bead5e989-dshm\") pod \"premium-w3-ce69f3-kserve-7bd85bd65-5z7vf\" (UID: \"583dd7dc-b160-468a-8ebe-781bead5e989\") " pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:26:12.442714 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:12.442690    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/583dd7dc-b160-468a-8ebe-781bead5e989-tmp-dir\") pod \"premium-w3-ce69f3-kserve-7bd85bd65-5z7vf\" (UID: \"583dd7dc-b160-468a-8ebe-781bead5e989\") " pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:26:12.442815 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:12.442794    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/583dd7dc-b160-468a-8ebe-781bead5e989-model-cache\") pod \"premium-w3-ce69f3-kserve-7bd85bd65-5z7vf\" (UID: \"583dd7dc-b160-468a-8ebe-781bead5e989\") " pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:26:12.443002 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:12.442977    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/583dd7dc-b160-468a-8ebe-781bead5e989-home\") pod \"premium-w3-ce69f3-kserve-7bd85bd65-5z7vf\" (UID: \"583dd7dc-b160-468a-8ebe-781bead5e989\") " pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:26:12.445252 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:12.445224    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/583dd7dc-b160-468a-8ebe-781bead5e989-dshm\") pod \"premium-w3-ce69f3-kserve-7bd85bd65-5z7vf\" (UID: \"583dd7dc-b160-468a-8ebe-781bead5e989\") " pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:26:12.445378 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:12.445235    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/583dd7dc-b160-468a-8ebe-781bead5e989-tls-certs\") pod \"premium-w3-ce69f3-kserve-7bd85bd65-5z7vf\" (UID: \"583dd7dc-b160-468a-8ebe-781bead5e989\") " pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:26:12.451455 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:12.451433    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lsp9\" (UniqueName: \"kubernetes.io/projected/583dd7dc-b160-468a-8ebe-781bead5e989-kube-api-access-5lsp9\") pod \"premium-w3-ce69f3-kserve-7bd85bd65-5z7vf\" (UID: \"583dd7dc-b160-468a-8ebe-781bead5e989\") " pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:26:12.549815 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:12.548986    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:26:12.838457 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:12.837532    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f"
Oct 30 19:26:12.838633 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:12.838464    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f"
Oct 30 19:26:12.844615 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:12.843362    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f"
Oct 30 19:26:12.885123 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:12.883900    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:26:12.904239 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:12.904202    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:26:12.963278 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:12.963245    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:26:13.024470 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:13.024432    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"]
Oct 30 19:26:13.043138 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:26:13.043087    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod583dd7dc_b160_468a_8ebe_781bead5e989.slice/crio-5a86424bfb321ba734bd024945cce8194549fd6cb9273025cc4eab5735d6c587 WatchSource:0}: Error finding container 5a86424bfb321ba734bd024945cce8194549fd6cb9273025cc4eab5735d6c587: Status 404 returned error can't find the container with id 5a86424bfb321ba734bd024945cce8194549fd6cb9273025cc4eab5735d6c587
Oct 30 19:26:13.955471 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:13.955424    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" event={"ID":"583dd7dc-b160-468a-8ebe-781bead5e989","Type":"ContainerStarted","Data":"3ab96f67f367f04f33b41a5298cb888f97257541d7ca8328a7c46a7dd1821d83"}
Oct 30 19:26:13.955917 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:13.955485    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" event={"ID":"583dd7dc-b160-468a-8ebe-781bead5e989","Type":"ContainerStarted","Data":"5a86424bfb321ba734bd024945cce8194549fd6cb9273025cc4eab5735d6c587"}
Oct 30 19:26:13.979092 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:13.979042    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podStartSLOduration=1.979027506 podStartE2EDuration="1.979027506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:13.978928533 +0000 UTC m=+593.489988501" watchObservedRunningTime="2026-10-30 19:26:13.979027506 +0000 UTC m=+593.490087470"
Oct 30 19:26:14.549480 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:14.549441    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:26:17.728538 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:17.728501    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"]
Oct 30 19:26:17.732837 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:17.732816    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:26:17.745084 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:17.744969    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"]
Oct 30 19:26:17.746461 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:17.746432    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b623ca\"/\"premium-w2-b623ca-kserve-self-signed-certs\""
Oct 30 19:26:17.822178 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:17.822133    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39590583-81fd-4e35-9047-ebd7a47b4fce-tmp-dir\") pod \"premium-w2-b623ca-kserve-55dcdc76cc-pjqrg\" (UID: \"39590583-81fd-4e35-9047-ebd7a47b4fce\") " pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:26:17.822375 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:17.822191    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7twl\" (UniqueName: \"kubernetes.io/projected/39590583-81fd-4e35-9047-ebd7a47b4fce-kube-api-access-j7twl\") pod \"premium-w2-b623ca-kserve-55dcdc76cc-pjqrg\" (UID: \"39590583-81fd-4e35-9047-ebd7a47b4fce\") " pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:26:17.822375 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:17.822334    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39590583-81fd-4e35-9047-ebd7a47b4fce-model-cache\") pod \"premium-w2-b623ca-kserve-55dcdc76cc-pjqrg\" (UID: \"39590583-81fd-4e35-9047-ebd7a47b4fce\") " pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:26:17.822375 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:17.822372    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39590583-81fd-4e35-9047-ebd7a47b4fce-tls-certs\") pod \"premium-w2-b623ca-kserve-55dcdc76cc-pjqrg\" (UID: \"39590583-81fd-4e35-9047-ebd7a47b4fce\") " pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:26:17.822563 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:17.822472    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39590583-81fd-4e35-9047-ebd7a47b4fce-home\") pod \"premium-w2-b623ca-kserve-55dcdc76cc-pjqrg\" (UID: \"39590583-81fd-4e35-9047-ebd7a47b4fce\") " pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:26:17.822563 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:17.822510    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39590583-81fd-4e35-9047-ebd7a47b4fce-dshm\") pod \"premium-w2-b623ca-kserve-55dcdc76cc-pjqrg\" (UID: \"39590583-81fd-4e35-9047-ebd7a47b4fce\") " pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:26:17.923530 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:17.923494    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39590583-81fd-4e35-9047-ebd7a47b4fce-tmp-dir\") pod \"premium-w2-b623ca-kserve-55dcdc76cc-pjqrg\" (UID: \"39590583-81fd-4e35-9047-ebd7a47b4fce\") " pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:26:17.924354 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:17.924330    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7twl\" (UniqueName: \"kubernetes.io/projected/39590583-81fd-4e35-9047-ebd7a47b4fce-kube-api-access-j7twl\") pod \"premium-w2-b623ca-kserve-55dcdc76cc-pjqrg\" (UID: \"39590583-81fd-4e35-9047-ebd7a47b4fce\") " pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:26:17.924961 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:17.924936    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39590583-81fd-4e35-9047-ebd7a47b4fce-model-cache\") pod \"premium-w2-b623ca-kserve-55dcdc76cc-pjqrg\" (UID: \"39590583-81fd-4e35-9047-ebd7a47b4fce\") " pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:26:17.930873 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:17.930837    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39590583-81fd-4e35-9047-ebd7a47b4fce-tls-certs\") pod \"premium-w2-b623ca-kserve-55dcdc76cc-pjqrg\" (UID: \"39590583-81fd-4e35-9047-ebd7a47b4fce\") " pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:26:17.931007 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:17.930934    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39590583-81fd-4e35-9047-ebd7a47b4fce-home\") pod \"premium-w2-b623ca-kserve-55dcdc76cc-pjqrg\" (UID: \"39590583-81fd-4e35-9047-ebd7a47b4fce\") " pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:26:17.931007 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:17.931000    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39590583-81fd-4e35-9047-ebd7a47b4fce-dshm\") pod \"premium-w2-b623ca-kserve-55dcdc76cc-pjqrg\" (UID: \"39590583-81fd-4e35-9047-ebd7a47b4fce\") " pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:26:17.934238 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:17.934216    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39590583-81fd-4e35-9047-ebd7a47b4fce-tls-certs\") pod \"premium-w2-b623ca-kserve-55dcdc76cc-pjqrg\" (UID: \"39590583-81fd-4e35-9047-ebd7a47b4fce\") " pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:26:17.934348 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:17.924231    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39590583-81fd-4e35-9047-ebd7a47b4fce-tmp-dir\") pod \"premium-w2-b623ca-kserve-55dcdc76cc-pjqrg\" (UID: \"39590583-81fd-4e35-9047-ebd7a47b4fce\") " pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:26:17.934348 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:17.925365    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39590583-81fd-4e35-9047-ebd7a47b4fce-model-cache\") pod \"premium-w2-b623ca-kserve-55dcdc76cc-pjqrg\" (UID: \"39590583-81fd-4e35-9047-ebd7a47b4fce\") " pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:26:17.934541 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:17.934523    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39590583-81fd-4e35-9047-ebd7a47b4fce-home\") pod \"premium-w2-b623ca-kserve-55dcdc76cc-pjqrg\" (UID: \"39590583-81fd-4e35-9047-ebd7a47b4fce\") " pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:26:17.944360 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:17.944330    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39590583-81fd-4e35-9047-ebd7a47b4fce-dshm\") pod \"premium-w2-b623ca-kserve-55dcdc76cc-pjqrg\" (UID: \"39590583-81fd-4e35-9047-ebd7a47b4fce\") " pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:26:17.954013 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:17.953984    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7twl\" (UniqueName: \"kubernetes.io/projected/39590583-81fd-4e35-9047-ebd7a47b4fce-kube-api-access-j7twl\") pod \"premium-w2-b623ca-kserve-55dcdc76cc-pjqrg\" (UID: \"39590583-81fd-4e35-9047-ebd7a47b4fce\") " pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:26:18.055145 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:18.055065    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:26:18.358170 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:26:18.358122    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39590583_81fd_4e35_9047_ebd7a47b4fce.slice/crio-cbcaa5a305e8d60dbd249f5adb24893e54d9b9afdfa13f7b35424a387fa3969c WatchSource:0}: Error finding container cbcaa5a305e8d60dbd249f5adb24893e54d9b9afdfa13f7b35424a387fa3969c: Status 404 returned error can't find the container with id cbcaa5a305e8d60dbd249f5adb24893e54d9b9afdfa13f7b35424a387fa3969c
Oct 30 19:26:18.367990 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:18.367960    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"]
Oct 30 19:26:18.989055 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:18.989013    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" event={"ID":"39590583-81fd-4e35-9047-ebd7a47b4fce","Type":"ContainerStarted","Data":"40f0dca84100a818eec5a1f5fe6451dd70e441f3405c85449d8d4942b7b8ec03"}
Oct 30 19:26:18.989470 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:18.989065    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" event={"ID":"39590583-81fd-4e35-9047-ebd7a47b4fce","Type":"ContainerStarted","Data":"cbcaa5a305e8d60dbd249f5adb24893e54d9b9afdfa13f7b35424a387fa3969c"}
Oct 30 19:26:19.009252 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:19.009204    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podStartSLOduration=2.009188496 podStartE2EDuration="2.009188496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:19.007128524 +0000 UTC m=+598.518188495" watchObservedRunningTime="2026-10-30 19:26:19.009188496 +0000 UTC m=+598.520248468"
Oct 30 19:26:19.056257 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:19.056226    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:26:20.703694 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:20.703669    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cmw6w_76e21916-0744-4499-a0cb-64a7a44a8285/console-operator/2.log"
Oct 30 19:26:20.704514 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:20.704491    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cmw6w_76e21916-0744-4499-a0cb-64a7a44a8285/console-operator/2.log"
Oct 30 19:26:22.550161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:22.550124    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:26:22.563209 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:22.563180    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:26:23.031595 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:23.031559    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:26:24.984288 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:26:24.984244    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:26:28.055915 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:28.055883    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:26:28.068993 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:28.068966    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:26:29.068091 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:29.068053    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:26:32.444205 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:32.444174    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"]
Oct 30 19:26:32.448144 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:32.448124    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:26:32.458798 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:32.458772    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"]
Oct 30 19:26:32.460043 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:32.460016    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ae63bf\"/\"premium-w0-ae63bf-kserve-self-signed-certs\""
Oct 30 19:26:32.498819 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:32.498793    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db0f2ea4-b059-4f08-8cdd-a27eddaac604-dshm\") pod \"premium-w0-ae63bf-kserve-865c7d559c-shtjb\" (UID: \"db0f2ea4-b059-4f08-8cdd-a27eddaac604\") " pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:26:32.498956 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:32.498838    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db0f2ea4-b059-4f08-8cdd-a27eddaac604-home\") pod \"premium-w0-ae63bf-kserve-865c7d559c-shtjb\" (UID: \"db0f2ea4-b059-4f08-8cdd-a27eddaac604\") " pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:26:32.498956 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:32.498856    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26tj6\" (UniqueName: \"kubernetes.io/projected/db0f2ea4-b059-4f08-8cdd-a27eddaac604-kube-api-access-26tj6\") pod \"premium-w0-ae63bf-kserve-865c7d559c-shtjb\" (UID: \"db0f2ea4-b059-4f08-8cdd-a27eddaac604\") " pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:26:32.499042 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:32.498952    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db0f2ea4-b059-4f08-8cdd-a27eddaac604-tmp-dir\") pod \"premium-w0-ae63bf-kserve-865c7d559c-shtjb\" (UID: \"db0f2ea4-b059-4f08-8cdd-a27eddaac604\") " pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:26:32.499042 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:32.499016    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db0f2ea4-b059-4f08-8cdd-a27eddaac604-tls-certs\") pod \"premium-w0-ae63bf-kserve-865c7d559c-shtjb\" (UID: \"db0f2ea4-b059-4f08-8cdd-a27eddaac604\") " pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:26:32.499125 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:32.499112    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db0f2ea4-b059-4f08-8cdd-a27eddaac604-model-cache\") pod \"premium-w0-ae63bf-kserve-865c7d559c-shtjb\" (UID: \"db0f2ea4-b059-4f08-8cdd-a27eddaac604\") " pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:26:32.600649 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:32.600614    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db0f2ea4-b059-4f08-8cdd-a27eddaac604-dshm\") pod \"premium-w0-ae63bf-kserve-865c7d559c-shtjb\" (UID: \"db0f2ea4-b059-4f08-8cdd-a27eddaac604\") " pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:26:32.600838 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:32.600676    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db0f2ea4-b059-4f08-8cdd-a27eddaac604-home\") pod \"premium-w0-ae63bf-kserve-865c7d559c-shtjb\" (UID: \"db0f2ea4-b059-4f08-8cdd-a27eddaac604\") " pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:26:32.600838 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:32.600705    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26tj6\" (UniqueName: \"kubernetes.io/projected/db0f2ea4-b059-4f08-8cdd-a27eddaac604-kube-api-access-26tj6\") pod \"premium-w0-ae63bf-kserve-865c7d559c-shtjb\" (UID: \"db0f2ea4-b059-4f08-8cdd-a27eddaac604\") " pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:26:32.600838 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:32.600763    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db0f2ea4-b059-4f08-8cdd-a27eddaac604-tmp-dir\") pod \"premium-w0-ae63bf-kserve-865c7d559c-shtjb\" (UID: \"db0f2ea4-b059-4f08-8cdd-a27eddaac604\") " pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:26:32.600838 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:32.600797    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db0f2ea4-b059-4f08-8cdd-a27eddaac604-tls-certs\") pod \"premium-w0-ae63bf-kserve-865c7d559c-shtjb\" (UID: \"db0f2ea4-b059-4f08-8cdd-a27eddaac604\") " pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:26:32.601084 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:32.600886    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db0f2ea4-b059-4f08-8cdd-a27eddaac604-model-cache\") pod \"premium-w0-ae63bf-kserve-865c7d559c-shtjb\" (UID: \"db0f2ea4-b059-4f08-8cdd-a27eddaac604\") " pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:26:32.601251 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:32.601223    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db0f2ea4-b059-4f08-8cdd-a27eddaac604-home\") pod \"premium-w0-ae63bf-kserve-865c7d559c-shtjb\" (UID: \"db0f2ea4-b059-4f08-8cdd-a27eddaac604\") " pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:26:32.601321 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:32.601234    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db0f2ea4-b059-4f08-8cdd-a27eddaac604-tmp-dir\") pod \"premium-w0-ae63bf-kserve-865c7d559c-shtjb\" (UID: \"db0f2ea4-b059-4f08-8cdd-a27eddaac604\") " pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:26:32.601321 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:32.601283    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db0f2ea4-b059-4f08-8cdd-a27eddaac604-model-cache\") pod \"premium-w0-ae63bf-kserve-865c7d559c-shtjb\" (UID: \"db0f2ea4-b059-4f08-8cdd-a27eddaac604\") " pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:26:32.603235 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:32.603198    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db0f2ea4-b059-4f08-8cdd-a27eddaac604-dshm\") pod \"premium-w0-ae63bf-kserve-865c7d559c-shtjb\" (UID: \"db0f2ea4-b059-4f08-8cdd-a27eddaac604\") " pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:26:32.603645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:32.603624    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db0f2ea4-b059-4f08-8cdd-a27eddaac604-tls-certs\") pod \"premium-w0-ae63bf-kserve-865c7d559c-shtjb\" (UID: \"db0f2ea4-b059-4f08-8cdd-a27eddaac604\") " pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:26:32.612471 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:32.612437    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26tj6\" (UniqueName: \"kubernetes.io/projected/db0f2ea4-b059-4f08-8cdd-a27eddaac604-kube-api-access-26tj6\") pod \"premium-w0-ae63bf-kserve-865c7d559c-shtjb\" (UID: \"db0f2ea4-b059-4f08-8cdd-a27eddaac604\") " pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:26:32.766003 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:32.765973    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:26:33.056551 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:33.056527    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"]
Oct 30 19:26:33.058892 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:26:33.058860    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb0f2ea4_b059_4f08_8cdd_a27eddaac604.slice/crio-5b5e9accbd4bf49065bef0d541a817db9cdc007aa07755297df3b94b6af3beb3 WatchSource:0}: Error finding container 5b5e9accbd4bf49065bef0d541a817db9cdc007aa07755297df3b94b6af3beb3: Status 404 returned error can't find the container with id 5b5e9accbd4bf49065bef0d541a817db9cdc007aa07755297df3b94b6af3beb3
Oct 30 19:26:33.098200 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:33.098160    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" event={"ID":"db0f2ea4-b059-4f08-8cdd-a27eddaac604","Type":"ContainerStarted","Data":"5b5e9accbd4bf49065bef0d541a817db9cdc007aa07755297df3b94b6af3beb3"}
Oct 30 19:26:34.106363 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:34.106327    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" event={"ID":"db0f2ea4-b059-4f08-8cdd-a27eddaac604","Type":"ContainerStarted","Data":"df48a93b78fe9651bdd1f4ecea80c1b57e74362edcc9e9c789bf3de4a9ed8ac3"}
Oct 30 19:26:34.126390 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:34.126334    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podStartSLOduration=2.126314967 podStartE2EDuration="2.126314967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:34.124115649 +0000 UTC m=+613.635175614" watchObservedRunningTime="2026-10-30 19:26:34.126314967 +0000 UTC m=+613.637374922"
Oct 30 19:26:34.766362 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:34.766328    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:26:42.766534 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:42.766486    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:26:42.779755 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:42.779711    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:26:43.176524 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:26:43.176425    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:26:54.979431 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:26:54.979389    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:27:21.110503 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:21.110464    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48c6bd74-vptpn"]
Oct 30 19:27:21.116330 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:21.116311    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48c6bd74-vptpn"
Oct 30 19:27:21.120013 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:21.119982    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:27:21.120931 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:21.120908    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:27:21.121267 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:21.121243    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:27:21.121427 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:21.121413    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:27:21.122370 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:21.122347    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48c6bd74-vptpn"]
Oct 30 19:27:21.174846 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:21.174718    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54cdd42b-39c3-4d86-bd89-13e49b1a2aed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48c6bd74-vptpn\" (UID: \"54cdd42b-39c3-4d86-bd89-13e49b1a2aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48c6bd74-vptpn"
Oct 30 19:27:21.175047 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:21.174944    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnlqd\" (UniqueName: \"kubernetes.io/projected/54cdd42b-39c3-4d86-bd89-13e49b1a2aed-kube-api-access-jnlqd\") pod \"maas-api-revoke-sub-48c6bd74-vptpn\" (UID: \"54cdd42b-39c3-4d86-bd89-13e49b1a2aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48c6bd74-vptpn"
Oct 30 19:27:21.175047 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:21.174991    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54cdd42b-39c3-4d86-bd89-13e49b1a2aed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48c6bd74-vptpn\" (UID: \"54cdd42b-39c3-4d86-bd89-13e49b1a2aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48c6bd74-vptpn"
Oct 30 19:27:21.276553 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:21.276518    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnlqd\" (UniqueName: \"kubernetes.io/projected/54cdd42b-39c3-4d86-bd89-13e49b1a2aed-kube-api-access-jnlqd\") pod \"maas-api-revoke-sub-48c6bd74-vptpn\" (UID: \"54cdd42b-39c3-4d86-bd89-13e49b1a2aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48c6bd74-vptpn"
Oct 30 19:27:21.276785 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:21.276565    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54cdd42b-39c3-4d86-bd89-13e49b1a2aed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48c6bd74-vptpn\" (UID: \"54cdd42b-39c3-4d86-bd89-13e49b1a2aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48c6bd74-vptpn"
Oct 30 19:27:21.276785 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:21.276642    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54cdd42b-39c3-4d86-bd89-13e49b1a2aed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48c6bd74-vptpn\" (UID: \"54cdd42b-39c3-4d86-bd89-13e49b1a2aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48c6bd74-vptpn"
Oct 30 19:27:21.277406 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:21.277370    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54cdd42b-39c3-4d86-bd89-13e49b1a2aed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48c6bd74-vptpn\" (UID: \"54cdd42b-39c3-4d86-bd89-13e49b1a2aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48c6bd74-vptpn"
Oct 30 19:27:21.279199 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:21.279180    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54cdd42b-39c3-4d86-bd89-13e49b1a2aed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48c6bd74-vptpn\" (UID: \"54cdd42b-39c3-4d86-bd89-13e49b1a2aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48c6bd74-vptpn"
Oct 30 19:27:21.286016 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:21.285996    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnlqd\" (UniqueName: \"kubernetes.io/projected/54cdd42b-39c3-4d86-bd89-13e49b1a2aed-kube-api-access-jnlqd\") pod \"maas-api-revoke-sub-48c6bd74-vptpn\" (UID: \"54cdd42b-39c3-4d86-bd89-13e49b1a2aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48c6bd74-vptpn"
Oct 30 19:27:21.434882 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:21.434792    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48c6bd74-vptpn"
Oct 30 19:27:21.716784 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:21.716758    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48c6bd74-vptpn"]
Oct 30 19:27:21.718766 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:27:21.718695    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54cdd42b_39c3_4d86_bd89_13e49b1a2aed.slice/crio-7ae1e65c31018554fcbba91e669063171b42f40013755fb7b35a8eb4e28b9d48 WatchSource:0}: Error finding container 7ae1e65c31018554fcbba91e669063171b42f40013755fb7b35a8eb4e28b9d48: Status 404 returned error can't find the container with id 7ae1e65c31018554fcbba91e669063171b42f40013755fb7b35a8eb4e28b9d48
Oct 30 19:27:22.293038 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:22.292988    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56482a81-gpcjk"]
Oct 30 19:27:22.312122 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:22.312097    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56482a81-gpcjk"
Oct 30 19:27:22.321991 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:22.321065    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56482a81-gpcjk"]
Oct 30 19:27:22.389856 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:22.389819    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bb701c3-0f12-4a31-bc50-ee5c96da70c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56482a81-gpcjk\" (UID: \"7bb701c3-0f12-4a31-bc50-ee5c96da70c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56482a81-gpcjk"
Oct 30 19:27:22.390047 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:22.389883    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bb701c3-0f12-4a31-bc50-ee5c96da70c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56482a81-gpcjk\" (UID: \"7bb701c3-0f12-4a31-bc50-ee5c96da70c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56482a81-gpcjk"
Oct 30 19:27:22.390117 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:22.390067    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzg7l\" (UniqueName: \"kubernetes.io/projected/7bb701c3-0f12-4a31-bc50-ee5c96da70c8-kube-api-access-mzg7l\") pod \"maas-api-revoke-sub-56482a81-gpcjk\" (UID: \"7bb701c3-0f12-4a31-bc50-ee5c96da70c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56482a81-gpcjk"
Oct 30 19:27:22.482076 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:22.482043    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48c6bd74-vptpn" event={"ID":"54cdd42b-39c3-4d86-bd89-13e49b1a2aed","Type":"ContainerStarted","Data":"7ae1e65c31018554fcbba91e669063171b42f40013755fb7b35a8eb4e28b9d48"}
Oct 30 19:27:22.491608 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:22.491578    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzg7l\" (UniqueName: \"kubernetes.io/projected/7bb701c3-0f12-4a31-bc50-ee5c96da70c8-kube-api-access-mzg7l\") pod \"maas-api-revoke-sub-56482a81-gpcjk\" (UID: \"7bb701c3-0f12-4a31-bc50-ee5c96da70c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56482a81-gpcjk"
Oct 30 19:27:22.491756 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:22.491684    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bb701c3-0f12-4a31-bc50-ee5c96da70c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56482a81-gpcjk\" (UID: \"7bb701c3-0f12-4a31-bc50-ee5c96da70c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56482a81-gpcjk"
Oct 30 19:27:22.491804 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:22.491730    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bb701c3-0f12-4a31-bc50-ee5c96da70c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56482a81-gpcjk\" (UID: \"7bb701c3-0f12-4a31-bc50-ee5c96da70c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56482a81-gpcjk"
Oct 30 19:27:22.492491 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:22.492426    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bb701c3-0f12-4a31-bc50-ee5c96da70c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56482a81-gpcjk\" (UID: \"7bb701c3-0f12-4a31-bc50-ee5c96da70c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56482a81-gpcjk"
Oct 30 19:27:22.494523 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:22.494499    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bb701c3-0f12-4a31-bc50-ee5c96da70c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56482a81-gpcjk\" (UID: \"7bb701c3-0f12-4a31-bc50-ee5c96da70c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56482a81-gpcjk"
Oct 30 19:27:22.499826 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:22.499803    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzg7l\" (UniqueName: \"kubernetes.io/projected/7bb701c3-0f12-4a31-bc50-ee5c96da70c8-kube-api-access-mzg7l\") pod \"maas-api-revoke-sub-56482a81-gpcjk\" (UID: \"7bb701c3-0f12-4a31-bc50-ee5c96da70c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56482a81-gpcjk"
Oct 30 19:27:22.639241 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:22.639157    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56482a81-gpcjk"
Oct 30 19:27:22.918981 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:22.918874    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56482a81-gpcjk"]
Oct 30 19:27:22.931932 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:27:22.931899    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bb701c3_0f12_4a31_bc50_ee5c96da70c8.slice/crio-64a9bf0ac1077dc67979d0052eccebb5127157e82dca9e73d22988e20369af1e WatchSource:0}: Error finding container 64a9bf0ac1077dc67979d0052eccebb5127157e82dca9e73d22988e20369af1e: Status 404 returned error can't find the container with id 64a9bf0ac1077dc67979d0052eccebb5127157e82dca9e73d22988e20369af1e
Oct 30 19:27:23.492372 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:23.492332    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56482a81-gpcjk" event={"ID":"7bb701c3-0f12-4a31-bc50-ee5c96da70c8","Type":"ContainerStarted","Data":"64a9bf0ac1077dc67979d0052eccebb5127157e82dca9e73d22988e20369af1e"}
Oct 30 19:27:24.710757 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:24.710438    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0bd04a4-9d7q2"]
Oct 30 19:27:24.724779 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:24.719990    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bd04a4-9d7q2"
Oct 30 19:27:24.724779 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:24.722822    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0bd04a4-9d7q2"]
Oct 30 19:27:24.866466 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:24.866425    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0bd04a4-9d7q2\" (UID: \"9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bd04a4-9d7q2"
Oct 30 19:27:24.866654 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:24.866566    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0bd04a4-9d7q2\" (UID: \"9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bd04a4-9d7q2"
Oct 30 19:27:24.866728 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:24.866699    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vw2b\" (UniqueName: \"kubernetes.io/projected/9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4-kube-api-access-9vw2b\") pod \"maas-api-revoke-sub-c0bd04a4-9d7q2\" (UID: \"9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bd04a4-9d7q2"
Oct 30 19:27:24.969198 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:24.968422    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0bd04a4-9d7q2\" (UID: \"9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bd04a4-9d7q2"
Oct 30 19:27:24.969198 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:24.968557    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vw2b\" (UniqueName: \"kubernetes.io/projected/9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4-kube-api-access-9vw2b\") pod \"maas-api-revoke-sub-c0bd04a4-9d7q2\" (UID: \"9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bd04a4-9d7q2"
Oct 30 19:27:24.969198 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:24.968675    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0bd04a4-9d7q2\" (UID: \"9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bd04a4-9d7q2"
Oct 30 19:27:24.970486 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:24.970180    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0bd04a4-9d7q2\" (UID: \"9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bd04a4-9d7q2"
Oct 30 19:27:24.971889 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:24.971851    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0bd04a4-9d7q2\" (UID: \"9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bd04a4-9d7q2"
Oct 30 19:27:24.979247 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:24.979198    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vw2b\" (UniqueName: \"kubernetes.io/projected/9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4-kube-api-access-9vw2b\") pod \"maas-api-revoke-sub-c0bd04a4-9d7q2\" (UID: \"9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bd04a4-9d7q2"
Oct 30 19:27:25.000080 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:27:24.999579    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:27:25.097890 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:25.097854    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bd04a4-9d7q2"
Oct 30 19:27:25.430759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:25.424090    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0bd04a4-9d7q2"]
Oct 30 19:27:25.528051 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:25.527972    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bd04a4-9d7q2" event={"ID":"9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4","Type":"ContainerStarted","Data":"b20d53b299ea279fdfa9c1748448d492b0533a8d32fbc31e28217d89c7612c0e"}
Oct 30 19:27:25.713071 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:25.711057    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-264d6e15-66mzm"]
Oct 30 19:27:25.783544 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:25.782630    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-264d6e15-66mzm"]
Oct 30 19:27:25.783544 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:25.782779    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264d6e15-66mzm"
Oct 30 19:27:25.803357 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:25.800433    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zjdx\" (UniqueName: \"kubernetes.io/projected/ee3b76e8-5f8e-4da4-b65c-04abab29e1dc-kube-api-access-2zjdx\") pod \"maas-api-revoke-sub-264d6e15-66mzm\" (UID: \"ee3b76e8-5f8e-4da4-b65c-04abab29e1dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264d6e15-66mzm"
Oct 30 19:27:25.805618 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:25.800594    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee3b76e8-5f8e-4da4-b65c-04abab29e1dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-264d6e15-66mzm\" (UID: \"ee3b76e8-5f8e-4da4-b65c-04abab29e1dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264d6e15-66mzm"
Oct 30 19:27:25.805618 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:25.805032    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee3b76e8-5f8e-4da4-b65c-04abab29e1dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-264d6e15-66mzm\" (UID: \"ee3b76e8-5f8e-4da4-b65c-04abab29e1dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264d6e15-66mzm"
Oct 30 19:27:25.906395 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:25.906362    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zjdx\" (UniqueName: \"kubernetes.io/projected/ee3b76e8-5f8e-4da4-b65c-04abab29e1dc-kube-api-access-2zjdx\") pod \"maas-api-revoke-sub-264d6e15-66mzm\" (UID: \"ee3b76e8-5f8e-4da4-b65c-04abab29e1dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264d6e15-66mzm"
Oct 30 19:27:25.906572 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:25.906495    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee3b76e8-5f8e-4da4-b65c-04abab29e1dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-264d6e15-66mzm\" (UID: \"ee3b76e8-5f8e-4da4-b65c-04abab29e1dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264d6e15-66mzm"
Oct 30 19:27:25.906572 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:25.906526    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee3b76e8-5f8e-4da4-b65c-04abab29e1dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-264d6e15-66mzm\" (UID: \"ee3b76e8-5f8e-4da4-b65c-04abab29e1dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264d6e15-66mzm"
Oct 30 19:27:25.908035 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:25.907970    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee3b76e8-5f8e-4da4-b65c-04abab29e1dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-264d6e15-66mzm\" (UID: \"ee3b76e8-5f8e-4da4-b65c-04abab29e1dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264d6e15-66mzm"
Oct 30 19:27:25.909506 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:25.909467    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee3b76e8-5f8e-4da4-b65c-04abab29e1dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-264d6e15-66mzm\" (UID: \"ee3b76e8-5f8e-4da4-b65c-04abab29e1dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264d6e15-66mzm"
Oct 30 19:27:25.918171 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:25.918119    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zjdx\" (UniqueName: \"kubernetes.io/projected/ee3b76e8-5f8e-4da4-b65c-04abab29e1dc-kube-api-access-2zjdx\") pod \"maas-api-revoke-sub-264d6e15-66mzm\" (UID: \"ee3b76e8-5f8e-4da4-b65c-04abab29e1dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264d6e15-66mzm"
Oct 30 19:27:26.119132 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:26.118886    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264d6e15-66mzm"
Oct 30 19:27:26.192437 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:26.192405    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bb3c919-fwrpr"]
Oct 30 19:27:26.201386 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:26.201359    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3c919-fwrpr"
Oct 30 19:27:26.204933 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:26.204908    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bb3c919-fwrpr"]
Oct 30 19:27:26.323391 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:26.317561    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjj5h\" (UniqueName: \"kubernetes.io/projected/e766a79e-dff8-4e7e-b6b5-b08f443b8c03-kube-api-access-fjj5h\") pod \"maas-api-revoke-sub-5bb3c919-fwrpr\" (UID: \"e766a79e-dff8-4e7e-b6b5-b08f443b8c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3c919-fwrpr"
Oct 30 19:27:26.323391 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:26.317955    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e766a79e-dff8-4e7e-b6b5-b08f443b8c03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bb3c919-fwrpr\" (UID: \"e766a79e-dff8-4e7e-b6b5-b08f443b8c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3c919-fwrpr"
Oct 30 19:27:26.323391 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:26.317992    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e766a79e-dff8-4e7e-b6b5-b08f443b8c03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bb3c919-fwrpr\" (UID: \"e766a79e-dff8-4e7e-b6b5-b08f443b8c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3c919-fwrpr"
Oct 30 19:27:26.419877 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:26.419844    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e766a79e-dff8-4e7e-b6b5-b08f443b8c03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bb3c919-fwrpr\" (UID: \"e766a79e-dff8-4e7e-b6b5-b08f443b8c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3c919-fwrpr"
Oct 30 19:27:26.420039 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:26.419908    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e766a79e-dff8-4e7e-b6b5-b08f443b8c03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bb3c919-fwrpr\" (UID: \"e766a79e-dff8-4e7e-b6b5-b08f443b8c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3c919-fwrpr"
Oct 30 19:27:26.420039 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:26.420014    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjj5h\" (UniqueName: \"kubernetes.io/projected/e766a79e-dff8-4e7e-b6b5-b08f443b8c03-kube-api-access-fjj5h\") pod \"maas-api-revoke-sub-5bb3c919-fwrpr\" (UID: \"e766a79e-dff8-4e7e-b6b5-b08f443b8c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3c919-fwrpr"
Oct 30 19:27:26.420571 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:26.420551    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e766a79e-dff8-4e7e-b6b5-b08f443b8c03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bb3c919-fwrpr\" (UID: \"e766a79e-dff8-4e7e-b6b5-b08f443b8c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3c919-fwrpr"
Oct 30 19:27:26.422847 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:26.422818    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e766a79e-dff8-4e7e-b6b5-b08f443b8c03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bb3c919-fwrpr\" (UID: \"e766a79e-dff8-4e7e-b6b5-b08f443b8c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3c919-fwrpr"
Oct 30 19:27:26.428192 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:26.428171    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjj5h\" (UniqueName: \"kubernetes.io/projected/e766a79e-dff8-4e7e-b6b5-b08f443b8c03-kube-api-access-fjj5h\") pod \"maas-api-revoke-sub-5bb3c919-fwrpr\" (UID: \"e766a79e-dff8-4e7e-b6b5-b08f443b8c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3c919-fwrpr"
Oct 30 19:27:26.473985 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:26.473954    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-264d6e15-66mzm"]
Oct 30 19:27:26.476535 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:27:26.476491    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee3b76e8_5f8e_4da4_b65c_04abab29e1dc.slice/crio-3d0d5a0a584702b4b423c638efd9d6854cb48e0e5e491f4bd308eb456b5bd03a WatchSource:0}: Error finding container 3d0d5a0a584702b4b423c638efd9d6854cb48e0e5e491f4bd308eb456b5bd03a: Status 404 returned error can't find the container with id 3d0d5a0a584702b4b423c638efd9d6854cb48e0e5e491f4bd308eb456b5bd03a
Oct 30 19:27:26.537471 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:26.537409    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264d6e15-66mzm" event={"ID":"ee3b76e8-5f8e-4da4-b65c-04abab29e1dc","Type":"ContainerStarted","Data":"3d0d5a0a584702b4b423c638efd9d6854cb48e0e5e491f4bd308eb456b5bd03a"}
Oct 30 19:27:26.543645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:26.543621    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3c919-fwrpr"
Oct 30 19:27:26.943747 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:26.943703    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bb3c919-fwrpr"]
Oct 30 19:27:26.945091 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:27:26.945062    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode766a79e_dff8_4e7e_b6b5_b08f443b8c03.slice/crio-8e6a21f726ae1843e94e67a99c8223eebdadc966433f2b03abc5bedf4898e213 WatchSource:0}: Error finding container 8e6a21f726ae1843e94e67a99c8223eebdadc966433f2b03abc5bedf4898e213: Status 404 returned error can't find the container with id 8e6a21f726ae1843e94e67a99c8223eebdadc966433f2b03abc5bedf4898e213
Oct 30 19:27:27.072436 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:27.072410    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t"]
Oct 30 19:27:27.078254 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:27.078217    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t"
Oct 30 19:27:27.084065 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:27.084038    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ae63bf-openshift-default-dockercfg-2n7kg\""
Oct 30 19:27:27.086655 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:27.086623    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t"]
Oct 30 19:27:27.131407 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:27.131371    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88d33c34-9387-4234-b953-446167850092-istio-token\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t\" (UID: \"88d33c34-9387-4234-b953-446167850092\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t"
Oct 30 19:27:27.131670 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:27.131647    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88d33c34-9387-4234-b953-446167850092-istio-podinfo\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t\" (UID: \"88d33c34-9387-4234-b953-446167850092\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t"
Oct 30 19:27:27.131855 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:27.131838    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88d33c34-9387-4234-b953-446167850092-workload-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t\" (UID: \"88d33c34-9387-4234-b953-446167850092\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t"
Oct 30 19:27:27.131974 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:27.131958    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88d33c34-9387-4234-b953-446167850092-istio-data\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t\" (UID: \"88d33c34-9387-4234-b953-446167850092\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t"
Oct 30 19:27:27.132109 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:27.132093    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88d33c34-9387-4234-b953-446167850092-credential-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t\" (UID: \"88d33c34-9387-4234-b953-446167850092\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t"
Oct 30 19:27:27.132263 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:27.132247    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88d33c34-9387-4234-b953-446167850092-istio-envoy\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t\" (UID: \"88d33c34-9387-4234-b953-446167850092\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t"
Oct 30 19:27:27.132454 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:27.132436    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88d33c34-9387-4234-b953-446167850092-workload-certs\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t\" (UID: \"88d33c34-9387-4234-b953-446167850092\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t"
Oct 30 19:27:27.132575 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:27.132559    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88d33c34-9387-4234-b953-446167850092-istiod-ca-cert\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t\" (UID: \"88d33c34-9387-4234-b953-446167850092\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t"
Oct 30 19:27:27.132707 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:27.132690    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cs47\" (UniqueName: \"kubernetes.io/projected/88d33c34-9387-4234-b953-446167850092-kube-api-access-2cs47\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t\" (UID: \"88d33c34-9387-4234-b953-446167850092\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t"
Oct 30 19:27:27.234307 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:27.234279    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88d33c34-9387-4234-b953-446167850092-workload-certs\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t\" (UID: \"88d33c34-9387-4234-b953-446167850092\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t"
Oct 30 19:27:27.234484 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:27.234318    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88d33c34-9387-4234-b953-446167850092-istiod-ca-cert\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t\" (UID: \"88d33c34-9387-4234-b953-446167850092\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t"
Oct 30 19:27:27.234484 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:27.234363    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cs47\" (UniqueName: \"kubernetes.io/projected/88d33c34-9387-4234-b953-446167850092-kube-api-access-2cs47\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t\" (UID: \"88d33c34-9387-4234-b953-446167850092\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t"
Oct 30 19:27:27.234484 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:27.234419    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88d33c34-9387-4234-b953-446167850092-istio-token\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t\" (UID: \"88d33c34-9387-4234-b953-446167850092\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t"
Oct 30 19:27:27.234484 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:27.234446    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88d33c34-9387-4234-b953-446167850092-istio-podinfo\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t\" (UID: \"88d33c34-9387-4234-b953-446167850092\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t"
Oct 30 19:27:27.234695 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:27.234496    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88d33c34-9387-4234-b953-446167850092-workload-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t\" (UID: \"88d33c34-9387-4234-b953-446167850092\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t"
Oct 30 19:27:27.234695 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:27.234522    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88d33c34-9387-4234-b953-446167850092-istio-data\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t\" (UID: \"88d33c34-9387-4234-b953-446167850092\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t"
Oct 30 19:27:27.234695 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:27.234578    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88d33c34-9387-4234-b953-446167850092-credential-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t\" (UID: \"88d33c34-9387-4234-b953-446167850092\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t"
Oct 30 19:27:27.234695 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:27.234658    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88d33c34-9387-4234-b953-446167850092-istio-envoy\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t\" (UID: \"88d33c34-9387-4234-b953-446167850092\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t"
Oct 30 19:27:27.235102 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:27.234970    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88d33c34-9387-4234-b953-446167850092-workload-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t\" (UID: \"88d33c34-9387-4234-b953-446167850092\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t"
Oct 30 19:27:27.235191 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:27.235167    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88d33c34-9387-4234-b953-446167850092-workload-certs\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t\" (UID: \"88d33c34-9387-4234-b953-446167850092\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t"
Oct 30 19:27:27.235251 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:27.235198    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88d33c34-9387-4234-b953-446167850092-istiod-ca-cert\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t\" (UID: \"88d33c34-9387-4234-b953-446167850092\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t"
Oct 30 19:27:27.235306 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:27.235244    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88d33c34-9387-4234-b953-446167850092-istio-data\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t\" (UID: \"88d33c34-9387-4234-b953-446167850092\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t"
Oct 30 19:27:27.235396 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:27.235377    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88d33c34-9387-4234-b953-446167850092-credential-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t\" (UID: \"88d33c34-9387-4234-b953-446167850092\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t"
Oct 30 19:27:27.258000 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:27.257226    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88d33c34-9387-4234-b953-446167850092-istio-envoy\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t\" (UID: \"88d33c34-9387-4234-b953-446167850092\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t"
Oct 30 19:27:27.258000 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:27.257562    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88d33c34-9387-4234-b953-446167850092-istio-podinfo\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t\" (UID: \"88d33c34-9387-4234-b953-446167850092\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t"
Oct 30 19:27:27.258000 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:27.257893    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88d33c34-9387-4234-b953-446167850092-istio-token\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t\" (UID: \"88d33c34-9387-4234-b953-446167850092\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t"
Oct 30 19:27:27.258000 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:27.257923    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cs47\" (UniqueName: \"kubernetes.io/projected/88d33c34-9387-4234-b953-446167850092-kube-api-access-2cs47\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t\" (UID: \"88d33c34-9387-4234-b953-446167850092\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t"
Oct 30 19:27:27.397243 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:27.397217    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t"
Oct 30 19:27:27.547524 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:27.547424    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bd04a4-9d7q2" event={"ID":"9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4","Type":"ContainerStarted","Data":"0e3745aaa570430063acdc892881882fe243c5e1d01b20454c13635a3130e9b8"}
Oct 30 19:27:27.549405 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:27.549369    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3c919-fwrpr" event={"ID":"e766a79e-dff8-4e7e-b6b5-b08f443b8c03","Type":"ContainerStarted","Data":"901f369a951a610ce2016f8d213d2c63629cab4f9bcaf497ed99281c383f6611"}
Oct 30 19:27:27.549555 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:27.549429    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3c919-fwrpr" event={"ID":"e766a79e-dff8-4e7e-b6b5-b08f443b8c03","Type":"ContainerStarted","Data":"8e6a21f726ae1843e94e67a99c8223eebdadc966433f2b03abc5bedf4898e213"}
Oct 30 19:27:27.551244 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:27.551214    5043 generic.go:356] "Generic (PLEG): container finished" podID="54cdd42b-39c3-4d86-bd89-13e49b1a2aed" containerID="777069f4d4eb11eeacc5860aa9bc839226e15109dd398d9f7d758896289eb7ba" exitCode=0
Oct 30 19:27:27.551353 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:27.551261    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48c6bd74-vptpn" event={"ID":"54cdd42b-39c3-4d86-bd89-13e49b1a2aed","Type":"ContainerDied","Data":"777069f4d4eb11eeacc5860aa9bc839226e15109dd398d9f7d758896289eb7ba"}
Oct 30 19:27:27.553212 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:27.553190    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264d6e15-66mzm" event={"ID":"ee3b76e8-5f8e-4da4-b65c-04abab29e1dc","Type":"ContainerStarted","Data":"2ba520ef9b55c000e0909de2296a3242fedc78015a21c5202485a2f4e35816f2"}
Oct 30 19:27:27.568699 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:27.568647    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bd04a4-9d7q2" podStartSLOduration=2.561738598 podStartE2EDuration="3.56862878s" totalImagesPullingTime="1.006890182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:27.563017007 +0000 UTC m=+667.074077053" watchObservedRunningTime="2026-10-30 19:27:27.56862878 +0000 UTC m=+667.079688746"
Oct 30 19:27:27.584790 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:27.582862    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3c919-fwrpr" podStartSLOduration=1.582844407 podStartE2EDuration="1.582844407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:27.579595658 +0000 UTC m=+667.090655970" watchObservedRunningTime="2026-10-30 19:27:27.582844407 +0000 UTC m=+667.093904373"
Oct 30 19:27:27.616779 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:27.616608    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264d6e15-66mzm" podStartSLOduration=2.6165916940000002 podStartE2EDuration="2.616591694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:27.611821173 +0000 UTC m=+667.122881139" watchObservedRunningTime="2026-10-30 19:27:27.616591694 +0000 UTC m=+667.127651660"
Oct 30 19:27:27.690780 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:27.690714    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t"]
Oct 30 19:27:27.696831 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:27:27.696799    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88d33c34_9387_4234_b953_446167850092.slice/crio-1612eb5a5bf67685dba8a0b903a054bb547030b8ad68eeacc0b016644b655e36 WatchSource:0}: Error finding container 1612eb5a5bf67685dba8a0b903a054bb547030b8ad68eeacc0b016644b655e36: Status 404 returned error can't find the container with id 1612eb5a5bf67685dba8a0b903a054bb547030b8ad68eeacc0b016644b655e36
Oct 30 19:27:27.699412 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:27.699375    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:27:27.699525 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:27.699458    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:27:27.699525 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:27.699500    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:27:28.577834 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:28.577375    5043 generic.go:356] "Generic (PLEG): container finished" podID="9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4" containerID="0e3745aaa570430063acdc892881882fe243c5e1d01b20454c13635a3130e9b8" exitCode=0
Oct 30 19:27:28.577834 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:28.577501    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bd04a4-9d7q2" event={"ID":"9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4","Type":"ContainerDied","Data":"0e3745aaa570430063acdc892881882fe243c5e1d01b20454c13635a3130e9b8"}
Oct 30 19:27:28.592758 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:28.587958    5043 generic.go:356] "Generic (PLEG): container finished" podID="7bb701c3-0f12-4a31-bc50-ee5c96da70c8" containerID="d3d85cf9cf4c584e50884d055554c89d3952816e08824ef0599589970d8fca5d" exitCode=0
Oct 30 19:27:28.592758 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:28.588096    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56482a81-gpcjk" event={"ID":"7bb701c3-0f12-4a31-bc50-ee5c96da70c8","Type":"ContainerDied","Data":"d3d85cf9cf4c584e50884d055554c89d3952816e08824ef0599589970d8fca5d"}
Oct 30 19:27:28.592758 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:28.591402    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t" event={"ID":"88d33c34-9387-4234-b953-446167850092","Type":"ContainerStarted","Data":"53a012b0030f36e1b61ed75f81affb415ca4eaa1f4182837fd73d5322d6c7752"}
Oct 30 19:27:28.592758 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:28.591458    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t" event={"ID":"88d33c34-9387-4234-b953-446167850092","Type":"ContainerStarted","Data":"1612eb5a5bf67685dba8a0b903a054bb547030b8ad68eeacc0b016644b655e36"}
Oct 30 19:27:28.601102 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:28.601069    5043 generic.go:356] "Generic (PLEG): container finished" podID="e766a79e-dff8-4e7e-b6b5-b08f443b8c03" containerID="901f369a951a610ce2016f8d213d2c63629cab4f9bcaf497ed99281c383f6611" exitCode=0
Oct 30 19:27:28.601286 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:28.601193    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3c919-fwrpr" event={"ID":"e766a79e-dff8-4e7e-b6b5-b08f443b8c03","Type":"ContainerDied","Data":"901f369a951a610ce2016f8d213d2c63629cab4f9bcaf497ed99281c383f6611"}
Oct 30 19:27:28.603069 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:28.603042    5043 generic.go:356] "Generic (PLEG): container finished" podID="ee3b76e8-5f8e-4da4-b65c-04abab29e1dc" containerID="2ba520ef9b55c000e0909de2296a3242fedc78015a21c5202485a2f4e35816f2" exitCode=0
Oct 30 19:27:28.603203 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:28.603092    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264d6e15-66mzm" event={"ID":"ee3b76e8-5f8e-4da4-b65c-04abab29e1dc","Type":"ContainerDied","Data":"2ba520ef9b55c000e0909de2296a3242fedc78015a21c5202485a2f4e35816f2"}
Oct 30 19:27:28.681760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:28.680917    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t" podStartSLOduration=1.680896176 podStartE2EDuration="1.680896176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:28.679230063 +0000 UTC m=+668.190290029" watchObservedRunningTime="2026-10-30 19:27:28.680896176 +0000 UTC m=+668.191956142"
Oct 30 19:27:28.848372 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:28.848348    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48c6bd74-vptpn"
Oct 30 19:27:28.906785 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:28.905959    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54cdd42b-39c3-4d86-bd89-13e49b1a2aed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54cdd42b-39c3-4d86-bd89-13e49b1a2aed-maas-api-openshift-ca\") pod \"54cdd42b-39c3-4d86-bd89-13e49b1a2aed\" (UID: \"54cdd42b-39c3-4d86-bd89-13e49b1a2aed\") "
Oct 30 19:27:28.906785 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:28.906138    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54cdd42b-39c3-4d86-bd89-13e49b1a2aed-kube-api-access-jnlqd\" (UniqueName: \"kubernetes.io/projected/54cdd42b-39c3-4d86-bd89-13e49b1a2aed-kube-api-access-jnlqd\") pod \"54cdd42b-39c3-4d86-bd89-13e49b1a2aed\" (UID: \"54cdd42b-39c3-4d86-bd89-13e49b1a2aed\") "
Oct 30 19:27:28.906785 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:28.906234    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54cdd42b-39c3-4d86-bd89-13e49b1a2aed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54cdd42b-39c3-4d86-bd89-13e49b1a2aed-maas-api-xks-ca\") pod \"54cdd42b-39c3-4d86-bd89-13e49b1a2aed\" (UID: \"54cdd42b-39c3-4d86-bd89-13e49b1a2aed\") "
Oct 30 19:27:28.906785 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:28.906436    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54cdd42b-39c3-4d86-bd89-13e49b1a2aed-maas-api-openshift-ca" pod "54cdd42b-39c3-4d86-bd89-13e49b1a2aed" (UID: "54cdd42b-39c3-4d86-bd89-13e49b1a2aed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:27:28.906785 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:28.906640    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54cdd42b-39c3-4d86-bd89-13e49b1a2aed-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:28.910898 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:28.910864    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54cdd42b-39c3-4d86-bd89-13e49b1a2aed-maas-api-xks-ca" pod "54cdd42b-39c3-4d86-bd89-13e49b1a2aed" (UID: "54cdd42b-39c3-4d86-bd89-13e49b1a2aed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:27:28.913393 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:28.913374    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54cdd42b-39c3-4d86-bd89-13e49b1a2aed-kube-api-access-jnlqd" pod "54cdd42b-39c3-4d86-bd89-13e49b1a2aed" (UID: "54cdd42b-39c3-4d86-bd89-13e49b1a2aed"). InnerVolumeSpecName "kube-api-access-jnlqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:27:29.008469 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:29.008433    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnlqd\" (UniqueName: \"kubernetes.io/projected/54cdd42b-39c3-4d86-bd89-13e49b1a2aed-kube-api-access-jnlqd\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:29.008469 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:29.008469    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54cdd42b-39c3-4d86-bd89-13e49b1a2aed-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:29.398088 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:29.398054    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t"
Oct 30 19:27:29.400304 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:29.400270    5043 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:27:29.400456 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:29.400332    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t" podUID="88d33c34-9387-4234-b953-446167850092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 19:27:29.611632 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:29.611600    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48c6bd74-vptpn"
Oct 30 19:27:29.612086 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:29.611629    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48c6bd74-vptpn" event={"ID":"54cdd42b-39c3-4d86-bd89-13e49b1a2aed","Type":"ContainerDied","Data":"7ae1e65c31018554fcbba91e669063171b42f40013755fb7b35a8eb4e28b9d48"}
Oct 30 19:27:29.612086 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:29.611695    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ae1e65c31018554fcbba91e669063171b42f40013755fb7b35a8eb4e28b9d48"
Oct 30 19:27:30.020839 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.020815    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3c919-fwrpr"
Oct 30 19:27:30.023776 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.023112    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264d6e15-66mzm"
Oct 30 19:27:30.031409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.031381    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee3b76e8-5f8e-4da4-b65c-04abab29e1dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee3b76e8-5f8e-4da4-b65c-04abab29e1dc-maas-api-openshift-ca\") pod \"ee3b76e8-5f8e-4da4-b65c-04abab29e1dc\" (UID: \"ee3b76e8-5f8e-4da4-b65c-04abab29e1dc\") "
Oct 30 19:27:30.031594 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.031576    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee3b76e8-5f8e-4da4-b65c-04abab29e1dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee3b76e8-5f8e-4da4-b65c-04abab29e1dc-maas-api-xks-ca\") pod \"ee3b76e8-5f8e-4da4-b65c-04abab29e1dc\" (UID: \"ee3b76e8-5f8e-4da4-b65c-04abab29e1dc\") "
Oct 30 19:27:30.031787 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.031767    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e766a79e-dff8-4e7e-b6b5-b08f443b8c03-kube-api-access-fjj5h\" (UniqueName: \"kubernetes.io/projected/e766a79e-dff8-4e7e-b6b5-b08f443b8c03-kube-api-access-fjj5h\") pod \"e766a79e-dff8-4e7e-b6b5-b08f443b8c03\" (UID: \"e766a79e-dff8-4e7e-b6b5-b08f443b8c03\") "
Oct 30 19:27:30.031924 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.031909    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e766a79e-dff8-4e7e-b6b5-b08f443b8c03-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e766a79e-dff8-4e7e-b6b5-b08f443b8c03-maas-api-openshift-ca\") pod \"e766a79e-dff8-4e7e-b6b5-b08f443b8c03\" (UID: \"e766a79e-dff8-4e7e-b6b5-b08f443b8c03\") "
Oct 30 19:27:30.032085 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.032069    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee3b76e8-5f8e-4da4-b65c-04abab29e1dc-kube-api-access-2zjdx\" (UniqueName: \"kubernetes.io/projected/ee3b76e8-5f8e-4da4-b65c-04abab29e1dc-kube-api-access-2zjdx\") pod \"ee3b76e8-5f8e-4da4-b65c-04abab29e1dc\" (UID: \"ee3b76e8-5f8e-4da4-b65c-04abab29e1dc\") "
Oct 30 19:27:30.032302 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.032285    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e766a79e-dff8-4e7e-b6b5-b08f443b8c03-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e766a79e-dff8-4e7e-b6b5-b08f443b8c03-maas-api-xks-ca\") pod \"e766a79e-dff8-4e7e-b6b5-b08f443b8c03\" (UID: \"e766a79e-dff8-4e7e-b6b5-b08f443b8c03\") "
Oct 30 19:27:30.039182 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.039155    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e766a79e-dff8-4e7e-b6b5-b08f443b8c03-maas-api-openshift-ca" pod "e766a79e-dff8-4e7e-b6b5-b08f443b8c03" (UID: "e766a79e-dff8-4e7e-b6b5-b08f443b8c03"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:27:30.040003 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.039978    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee3b76e8-5f8e-4da4-b65c-04abab29e1dc-maas-api-openshift-ca" pod "ee3b76e8-5f8e-4da4-b65c-04abab29e1dc" (UID: "ee3b76e8-5f8e-4da4-b65c-04abab29e1dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:27:30.041216 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.041189    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e766a79e-dff8-4e7e-b6b5-b08f443b8c03-kube-api-access-fjj5h" pod "e766a79e-dff8-4e7e-b6b5-b08f443b8c03" (UID: "e766a79e-dff8-4e7e-b6b5-b08f443b8c03"). InnerVolumeSpecName "kube-api-access-fjj5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:27:30.043811 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.043667    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e766a79e-dff8-4e7e-b6b5-b08f443b8c03-maas-api-xks-ca" pod "e766a79e-dff8-4e7e-b6b5-b08f443b8c03" (UID: "e766a79e-dff8-4e7e-b6b5-b08f443b8c03"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:27:30.043811 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.043685    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee3b76e8-5f8e-4da4-b65c-04abab29e1dc-kube-api-access-2zjdx" pod "ee3b76e8-5f8e-4da4-b65c-04abab29e1dc" (UID: "ee3b76e8-5f8e-4da4-b65c-04abab29e1dc"). InnerVolumeSpecName "kube-api-access-2zjdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:27:30.044669 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.044633    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee3b76e8-5f8e-4da4-b65c-04abab29e1dc-maas-api-xks-ca" pod "ee3b76e8-5f8e-4da4-b65c-04abab29e1dc" (UID: "ee3b76e8-5f8e-4da4-b65c-04abab29e1dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:27:30.134578 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.134538    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zjdx\" (UniqueName: \"kubernetes.io/projected/ee3b76e8-5f8e-4da4-b65c-04abab29e1dc-kube-api-access-2zjdx\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:30.134578 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.134579    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e766a79e-dff8-4e7e-b6b5-b08f443b8c03-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:30.134881 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.134595    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee3b76e8-5f8e-4da4-b65c-04abab29e1dc-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:30.134881 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.134608    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee3b76e8-5f8e-4da4-b65c-04abab29e1dc-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:30.134881 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.134622    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjj5h\" (UniqueName: \"kubernetes.io/projected/e766a79e-dff8-4e7e-b6b5-b08f443b8c03-kube-api-access-fjj5h\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:30.134881 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.134635    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e766a79e-dff8-4e7e-b6b5-b08f443b8c03-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:30.139777 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.137802    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56482a81-gpcjk"
Oct 30 19:27:30.139976 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.139955    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bd04a4-9d7q2"
Oct 30 19:27:30.235943 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.235614    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bb701c3-0f12-4a31-bc50-ee5c96da70c8-kube-api-access-mzg7l\" (UniqueName: \"kubernetes.io/projected/7bb701c3-0f12-4a31-bc50-ee5c96da70c8-kube-api-access-mzg7l\") pod \"7bb701c3-0f12-4a31-bc50-ee5c96da70c8\" (UID: \"7bb701c3-0f12-4a31-bc50-ee5c96da70c8\") "
Oct 30 19:27:30.235943 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.235689    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bb701c3-0f12-4a31-bc50-ee5c96da70c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bb701c3-0f12-4a31-bc50-ee5c96da70c8-maas-api-xks-ca\") pod \"7bb701c3-0f12-4a31-bc50-ee5c96da70c8\" (UID: \"7bb701c3-0f12-4a31-bc50-ee5c96da70c8\") "
Oct 30 19:27:30.235943 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.235893    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4-maas-api-xks-ca\") pod \"9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4\" (UID: \"9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4\") "
Oct 30 19:27:30.235943 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.235924    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4-kube-api-access-9vw2b\" (UniqueName: \"kubernetes.io/projected/9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4-kube-api-access-9vw2b\") pod \"9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4\" (UID: \"9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4\") "
Oct 30 19:27:30.236244 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.235975    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bb701c3-0f12-4a31-bc50-ee5c96da70c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bb701c3-0f12-4a31-bc50-ee5c96da70c8-maas-api-openshift-ca\") pod \"7bb701c3-0f12-4a31-bc50-ee5c96da70c8\" (UID: \"7bb701c3-0f12-4a31-bc50-ee5c96da70c8\") "
Oct 30 19:27:30.236244 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.236001    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4-maas-api-openshift-ca\") pod \"9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4\" (UID: \"9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4\") "
Oct 30 19:27:30.236979 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.236949    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bb701c3-0f12-4a31-bc50-ee5c96da70c8-maas-api-openshift-ca" pod "7bb701c3-0f12-4a31-bc50-ee5c96da70c8" (UID: "7bb701c3-0f12-4a31-bc50-ee5c96da70c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:27:30.237120 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.237071    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4-maas-api-openshift-ca" pod "9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4" (UID: "9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:27:30.240962 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.238772    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bb701c3-0f12-4a31-bc50-ee5c96da70c8-kube-api-access-mzg7l" pod "7bb701c3-0f12-4a31-bc50-ee5c96da70c8" (UID: "7bb701c3-0f12-4a31-bc50-ee5c96da70c8"). InnerVolumeSpecName "kube-api-access-mzg7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:27:30.240962 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.238930    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4-maas-api-xks-ca" pod "9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4" (UID: "9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:27:30.240962 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.239319    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4-kube-api-access-9vw2b" pod "9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4" (UID: "9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4"). InnerVolumeSpecName "kube-api-access-9vw2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:27:30.241255 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.240962    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bb701c3-0f12-4a31-bc50-ee5c96da70c8-maas-api-xks-ca" pod "7bb701c3-0f12-4a31-bc50-ee5c96da70c8" (UID: "7bb701c3-0f12-4a31-bc50-ee5c96da70c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:27:30.337276 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.337246    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzg7l\" (UniqueName: \"kubernetes.io/projected/7bb701c3-0f12-4a31-bc50-ee5c96da70c8-kube-api-access-mzg7l\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:30.337276 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.337274    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bb701c3-0f12-4a31-bc50-ee5c96da70c8-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:30.337276 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.337284    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:30.337509 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.337298    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vw2b\" (UniqueName: \"kubernetes.io/projected/9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4-kube-api-access-9vw2b\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:30.337509 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.337306    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bb701c3-0f12-4a31-bc50-ee5c96da70c8-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:30.337509 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.337317    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:30.398537 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.398203    5043 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:27:30.398537 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.398251    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t" podUID="88d33c34-9387-4234-b953-446167850092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 19:27:30.619897 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.619810    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56482a81-gpcjk"
Oct 30 19:27:30.620361 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.619884    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56482a81-gpcjk" event={"ID":"7bb701c3-0f12-4a31-bc50-ee5c96da70c8","Type":"ContainerDied","Data":"64a9bf0ac1077dc67979d0052eccebb5127157e82dca9e73d22988e20369af1e"}
Oct 30 19:27:30.620361 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.619935    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64a9bf0ac1077dc67979d0052eccebb5127157e82dca9e73d22988e20369af1e"
Oct 30 19:27:30.621416 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.621375    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3c919-fwrpr" event={"ID":"e766a79e-dff8-4e7e-b6b5-b08f443b8c03","Type":"ContainerDied","Data":"8e6a21f726ae1843e94e67a99c8223eebdadc966433f2b03abc5bedf4898e213"}
Oct 30 19:27:30.621416 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.621408    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3c919-fwrpr"
Oct 30 19:27:30.621619 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.621424    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e6a21f726ae1843e94e67a99c8223eebdadc966433f2b03abc5bedf4898e213"
Oct 30 19:27:30.622779 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.622763    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264d6e15-66mzm"
Oct 30 19:27:30.622881 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.622776    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264d6e15-66mzm" event={"ID":"ee3b76e8-5f8e-4da4-b65c-04abab29e1dc","Type":"ContainerDied","Data":"3d0d5a0a584702b4b423c638efd9d6854cb48e0e5e491f4bd308eb456b5bd03a"}
Oct 30 19:27:30.622881 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.622826    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d0d5a0a584702b4b423c638efd9d6854cb48e0e5e491f4bd308eb456b5bd03a"
Oct 30 19:27:30.624431 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.624410    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bd04a4-9d7q2"
Oct 30 19:27:30.624526 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.624431    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bd04a4-9d7q2" event={"ID":"9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4","Type":"ContainerDied","Data":"b20d53b299ea279fdfa9c1748448d492b0533a8d32fbc31e28217d89c7612c0e"}
Oct 30 19:27:30.624526 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:30.624473    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b20d53b299ea279fdfa9c1748448d492b0533a8d32fbc31e28217d89c7612c0e"
Oct 30 19:27:31.398034 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:31.397996    5043 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:27:31.398220 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:31.398062    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t" podUID="88d33c34-9387-4234-b953-446167850092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 19:27:32.415194 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:32.415155    5043 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:27:32.416778 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:32.416720    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t" podUID="88d33c34-9387-4234-b953-446167850092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 19:27:33.398767 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:33.397697    5043 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:27:33.398767 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:33.397779    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t" podUID="88d33c34-9387-4234-b953-446167850092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 19:27:33.747763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:33.745257    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:27:33.805100 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:33.804819    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:34.398330 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:34.398291    5043 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:27:34.398493 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:34.398360    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t" podUID="88d33c34-9387-4234-b953-446167850092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 19:27:34.537771 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:34.537708    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-921f1fba-wqgxb"]
Oct 30 19:27:34.538292 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:34.538276    5043 cpu_manager.go:414] "containerMap: removing container" podUID="ee3b76e8-5f8e-4da4-b65c-04abab29e1dc" containerName="revoke-keys"
Oct 30 19:27:34.538292 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:34.538293    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee3b76e8-5f8e-4da4-b65c-04abab29e1dc" containerName="revoke-keys"
Oct 30 19:27:34.538418 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:34.538302    5043 cpu_manager.go:414] "containerMap: removing container" podUID="7bb701c3-0f12-4a31-bc50-ee5c96da70c8" containerName="revoke-keys"
Oct 30 19:27:34.538418 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:34.538307    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bb701c3-0f12-4a31-bc50-ee5c96da70c8" containerName="revoke-keys"
Oct 30 19:27:34.538418 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:34.538345    5043 cpu_manager.go:414] "containerMap: removing container" podUID="54cdd42b-39c3-4d86-bd89-13e49b1a2aed" containerName="revoke-keys"
Oct 30 19:27:34.538418 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:34.538350    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54cdd42b-39c3-4d86-bd89-13e49b1a2aed" containerName="revoke-keys"
Oct 30 19:27:34.538418 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:34.538361    5043 cpu_manager.go:414] "containerMap: removing container" podUID="9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4" containerName="revoke-keys"
Oct 30 19:27:34.538418 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:34.538366    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4" containerName="revoke-keys"
Oct 30 19:27:34.538418 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:34.538373    5043 cpu_manager.go:414] "containerMap: removing container" podUID="e766a79e-dff8-4e7e-b6b5-b08f443b8c03" containerName="revoke-keys"
Oct 30 19:27:34.538418 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:34.538378    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e766a79e-dff8-4e7e-b6b5-b08f443b8c03" containerName="revoke-keys"
Oct 30 19:27:34.538872 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:34.538456    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="54cdd42b-39c3-4d86-bd89-13e49b1a2aed" containerName="revoke-keys"
Oct 30 19:27:34.538872 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:34.538465    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4" containerName="revoke-keys"
Oct 30 19:27:34.538872 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:34.538470    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="e766a79e-dff8-4e7e-b6b5-b08f443b8c03" containerName="revoke-keys"
Oct 30 19:27:34.538872 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:34.538478    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee3b76e8-5f8e-4da4-b65c-04abab29e1dc" containerName="revoke-keys"
Oct 30 19:27:34.538872 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:34.538483    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bb701c3-0f12-4a31-bc50-ee5c96da70c8" containerName="revoke-keys"
Oct 30 19:27:34.541758 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:34.541724    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-921f1fba-wqgxb"
Oct 30 19:27:34.547983 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:34.547325    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:27:34.547983 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:34.547391    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:27:34.547983 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:34.547472    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:27:34.547983 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:34.547523    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:27:34.551331 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:34.551307    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-921f1fba-wqgxb"]
Oct 30 19:27:34.630581 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:34.630550    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xthz2\" (UniqueName: \"kubernetes.io/projected/58817e5e-c488-42bb-8de7-76df0d2652fa-kube-api-access-xthz2\") pod \"maas-api-revoke-sub-921f1fba-wqgxb\" (UID: \"58817e5e-c488-42bb-8de7-76df0d2652fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-921f1fba-wqgxb"
Oct 30 19:27:34.630729 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:34.630590    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58817e5e-c488-42bb-8de7-76df0d2652fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-921f1fba-wqgxb\" (UID: \"58817e5e-c488-42bb-8de7-76df0d2652fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-921f1fba-wqgxb"
Oct 30 19:27:34.630848 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:34.630768    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58817e5e-c488-42bb-8de7-76df0d2652fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-921f1fba-wqgxb\" (UID: \"58817e5e-c488-42bb-8de7-76df0d2652fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-921f1fba-wqgxb"
Oct 30 19:27:34.731726 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:34.731694    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58817e5e-c488-42bb-8de7-76df0d2652fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-921f1fba-wqgxb\" (UID: \"58817e5e-c488-42bb-8de7-76df0d2652fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-921f1fba-wqgxb"
Oct 30 19:27:34.731923 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:34.731812    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xthz2\" (UniqueName: \"kubernetes.io/projected/58817e5e-c488-42bb-8de7-76df0d2652fa-kube-api-access-xthz2\") pod \"maas-api-revoke-sub-921f1fba-wqgxb\" (UID: \"58817e5e-c488-42bb-8de7-76df0d2652fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-921f1fba-wqgxb"
Oct 30 19:27:34.731923 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:34.731836    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58817e5e-c488-42bb-8de7-76df0d2652fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-921f1fba-wqgxb\" (UID: \"58817e5e-c488-42bb-8de7-76df0d2652fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-921f1fba-wqgxb"
Oct 30 19:27:34.732452 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:34.732429    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58817e5e-c488-42bb-8de7-76df0d2652fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-921f1fba-wqgxb\" (UID: \"58817e5e-c488-42bb-8de7-76df0d2652fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-921f1fba-wqgxb"
Oct 30 19:27:34.734205 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:34.734182    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58817e5e-c488-42bb-8de7-76df0d2652fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-921f1fba-wqgxb\" (UID: \"58817e5e-c488-42bb-8de7-76df0d2652fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-921f1fba-wqgxb"
Oct 30 19:27:34.740180 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:34.740156    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xthz2\" (UniqueName: \"kubernetes.io/projected/58817e5e-c488-42bb-8de7-76df0d2652fa-kube-api-access-xthz2\") pod \"maas-api-revoke-sub-921f1fba-wqgxb\" (UID: \"58817e5e-c488-42bb-8de7-76df0d2652fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-921f1fba-wqgxb"
Oct 30 19:27:34.865592 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:34.865551    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-921f1fba-wqgxb"
Oct 30 19:27:35.061975 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:35.061900    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4e57ba6-c6bq7"]
Oct 30 19:27:35.066641 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:35.066622    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e57ba6-c6bq7"
Oct 30 19:27:35.081271 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:35.081245    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4e57ba6-c6bq7"]
Oct 30 19:27:35.136281 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:35.136233    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52d38b83-2cbe-4e80-adae-f837d0c9115b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4e57ba6-c6bq7\" (UID: \"52d38b83-2cbe-4e80-adae-f837d0c9115b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e57ba6-c6bq7"
Oct 30 19:27:35.136416 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:35.136357    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52d38b83-2cbe-4e80-adae-f837d0c9115b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4e57ba6-c6bq7\" (UID: \"52d38b83-2cbe-4e80-adae-f837d0c9115b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e57ba6-c6bq7"
Oct 30 19:27:35.136470 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:35.136420    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgzvr\" (UniqueName: \"kubernetes.io/projected/52d38b83-2cbe-4e80-adae-f837d0c9115b-kube-api-access-sgzvr\") pod \"maas-api-revoke-sub-d4e57ba6-c6bq7\" (UID: \"52d38b83-2cbe-4e80-adae-f837d0c9115b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e57ba6-c6bq7"
Oct 30 19:27:35.156983 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:35.156957    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-921f1fba-wqgxb"]
Oct 30 19:27:35.159418 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:27:35.159394    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58817e5e_c488_42bb_8de7_76df0d2652fa.slice/crio-0b655d227a4061e7f248aa6b541c77b68f62682ad28b9cd2562740ee4fbe00cf WatchSource:0}: Error finding container 0b655d227a4061e7f248aa6b541c77b68f62682ad28b9cd2562740ee4fbe00cf: Status 404 returned error can't find the container with id 0b655d227a4061e7f248aa6b541c77b68f62682ad28b9cd2562740ee4fbe00cf
Oct 30 19:27:35.237141 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:35.237113    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52d38b83-2cbe-4e80-adae-f837d0c9115b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4e57ba6-c6bq7\" (UID: \"52d38b83-2cbe-4e80-adae-f837d0c9115b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e57ba6-c6bq7"
Oct 30 19:27:35.237272 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:35.237172    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52d38b83-2cbe-4e80-adae-f837d0c9115b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4e57ba6-c6bq7\" (UID: \"52d38b83-2cbe-4e80-adae-f837d0c9115b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e57ba6-c6bq7"
Oct 30 19:27:35.237272 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:35.237227    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgzvr\" (UniqueName: \"kubernetes.io/projected/52d38b83-2cbe-4e80-adae-f837d0c9115b-kube-api-access-sgzvr\") pod \"maas-api-revoke-sub-d4e57ba6-c6bq7\" (UID: \"52d38b83-2cbe-4e80-adae-f837d0c9115b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e57ba6-c6bq7"
Oct 30 19:27:35.237952 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:35.237929    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52d38b83-2cbe-4e80-adae-f837d0c9115b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4e57ba6-c6bq7\" (UID: \"52d38b83-2cbe-4e80-adae-f837d0c9115b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e57ba6-c6bq7"
Oct 30 19:27:35.239538 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:35.239520    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52d38b83-2cbe-4e80-adae-f837d0c9115b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4e57ba6-c6bq7\" (UID: \"52d38b83-2cbe-4e80-adae-f837d0c9115b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e57ba6-c6bq7"
Oct 30 19:27:35.245338 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:35.245315    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgzvr\" (UniqueName: \"kubernetes.io/projected/52d38b83-2cbe-4e80-adae-f837d0c9115b-kube-api-access-sgzvr\") pod \"maas-api-revoke-sub-d4e57ba6-c6bq7\" (UID: \"52d38b83-2cbe-4e80-adae-f837d0c9115b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e57ba6-c6bq7"
Oct 30 19:27:35.389437 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:35.389392    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e57ba6-c6bq7"
Oct 30 19:27:35.403683 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:35.403652    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t"
Oct 30 19:27:35.404185 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:35.404076    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t"
Oct 30 19:27:35.405074 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:35.405023    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t"
Oct 30 19:27:35.669523 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:35.669496    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4e57ba6-c6bq7"]
Oct 30 19:27:35.671543 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:27:35.671516    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52d38b83_2cbe_4e80_adae_f837d0c9115b.slice/crio-b7dc3afc9784330cd769d5f31c0052ae0c9c69bef2af6a0b5aacf2d3b21c7eb0 WatchSource:0}: Error finding container b7dc3afc9784330cd769d5f31c0052ae0c9c69bef2af6a0b5aacf2d3b21c7eb0: Status 404 returned error can't find the container with id b7dc3afc9784330cd769d5f31c0052ae0c9c69bef2af6a0b5aacf2d3b21c7eb0
Oct 30 19:27:35.788966 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:35.788298    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-921f1fba-wqgxb" event={"ID":"58817e5e-c488-42bb-8de7-76df0d2652fa","Type":"ContainerStarted","Data":"6ff7887b82f2093fa35a3a8e25dafdc854523d7a21407d568d9564348186ab13"}
Oct 30 19:27:35.788966 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:35.788362    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-921f1fba-wqgxb" event={"ID":"58817e5e-c488-42bb-8de7-76df0d2652fa","Type":"ContainerStarted","Data":"0b655d227a4061e7f248aa6b541c77b68f62682ad28b9cd2562740ee4fbe00cf"}
Oct 30 19:27:35.790402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:35.790373    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e57ba6-c6bq7" event={"ID":"52d38b83-2cbe-4e80-adae-f837d0c9115b","Type":"ContainerStarted","Data":"b7dc3afc9784330cd769d5f31c0052ae0c9c69bef2af6a0b5aacf2d3b21c7eb0"}
Oct 30 19:27:35.807253 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:35.807206    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-921f1fba-wqgxb" podStartSLOduration=1.807192363 podStartE2EDuration="1.807192363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:35.806157735 +0000 UTC m=+675.317217701" watchObservedRunningTime="2026-10-30 19:27:35.807192363 +0000 UTC m=+675.318252324"
Oct 30 19:27:36.431510 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:36.431435    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89f15f13-22hc6"]
Oct 30 19:27:36.435152 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:36.435135    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f15f13-22hc6"
Oct 30 19:27:36.443045 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:36.443021    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89f15f13-22hc6"]
Oct 30 19:27:36.451113 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:36.450131    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpdhx\" (UniqueName: \"kubernetes.io/projected/888f7b8b-5ffa-40b6-b428-ae77ee29c855-kube-api-access-kpdhx\") pod \"maas-api-revoke-sub-89f15f13-22hc6\" (UID: \"888f7b8b-5ffa-40b6-b428-ae77ee29c855\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f15f13-22hc6"
Oct 30 19:27:36.451113 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:36.450229    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/888f7b8b-5ffa-40b6-b428-ae77ee29c855-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89f15f13-22hc6\" (UID: \"888f7b8b-5ffa-40b6-b428-ae77ee29c855\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f15f13-22hc6"
Oct 30 19:27:36.451113 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:36.450341    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/888f7b8b-5ffa-40b6-b428-ae77ee29c855-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89f15f13-22hc6\" (UID: \"888f7b8b-5ffa-40b6-b428-ae77ee29c855\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f15f13-22hc6"
Oct 30 19:27:36.551186 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:36.551156    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/888f7b8b-5ffa-40b6-b428-ae77ee29c855-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89f15f13-22hc6\" (UID: \"888f7b8b-5ffa-40b6-b428-ae77ee29c855\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f15f13-22hc6"
Oct 30 19:27:36.551354 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:36.551249    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/888f7b8b-5ffa-40b6-b428-ae77ee29c855-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89f15f13-22hc6\" (UID: \"888f7b8b-5ffa-40b6-b428-ae77ee29c855\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f15f13-22hc6"
Oct 30 19:27:36.551354 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:36.551305    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpdhx\" (UniqueName: \"kubernetes.io/projected/888f7b8b-5ffa-40b6-b428-ae77ee29c855-kube-api-access-kpdhx\") pod \"maas-api-revoke-sub-89f15f13-22hc6\" (UID: \"888f7b8b-5ffa-40b6-b428-ae77ee29c855\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f15f13-22hc6"
Oct 30 19:27:36.551949 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:36.551922    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/888f7b8b-5ffa-40b6-b428-ae77ee29c855-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89f15f13-22hc6\" (UID: \"888f7b8b-5ffa-40b6-b428-ae77ee29c855\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f15f13-22hc6"
Oct 30 19:27:36.553675 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:36.553649    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/888f7b8b-5ffa-40b6-b428-ae77ee29c855-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89f15f13-22hc6\" (UID: \"888f7b8b-5ffa-40b6-b428-ae77ee29c855\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f15f13-22hc6"
Oct 30 19:27:36.560491 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:36.560466    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpdhx\" (UniqueName: \"kubernetes.io/projected/888f7b8b-5ffa-40b6-b428-ae77ee29c855-kube-api-access-kpdhx\") pod \"maas-api-revoke-sub-89f15f13-22hc6\" (UID: \"888f7b8b-5ffa-40b6-b428-ae77ee29c855\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f15f13-22hc6"
Oct 30 19:27:36.754921 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:36.754893    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f15f13-22hc6"
Oct 30 19:27:36.799868 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:36.799355    5043 generic.go:356] "Generic (PLEG): container finished" podID="58817e5e-c488-42bb-8de7-76df0d2652fa" containerID="6ff7887b82f2093fa35a3a8e25dafdc854523d7a21407d568d9564348186ab13" exitCode=0
Oct 30 19:27:36.799868 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:36.799479    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-921f1fba-wqgxb" event={"ID":"58817e5e-c488-42bb-8de7-76df0d2652fa","Type":"ContainerDied","Data":"6ff7887b82f2093fa35a3a8e25dafdc854523d7a21407d568d9564348186ab13"}
Oct 30 19:27:36.802023 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:36.801993    5043 generic.go:356] "Generic (PLEG): container finished" podID="52d38b83-2cbe-4e80-adae-f837d0c9115b" containerID="5073573f1a0f66acb54784b6f0c28f4568be3c3dfb324d957eb89bd0be74d590" exitCode=0
Oct 30 19:27:36.802906 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:36.802885    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e57ba6-c6bq7" event={"ID":"52d38b83-2cbe-4e80-adae-f837d0c9115b","Type":"ContainerDied","Data":"5073573f1a0f66acb54784b6f0c28f4568be3c3dfb324d957eb89bd0be74d590"}
Oct 30 19:27:37.075410 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:37.075386    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89f15f13-22hc6"]
Oct 30 19:27:37.075983 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:27:37.075957    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod888f7b8b_5ffa_40b6_b428_ae77ee29c855.slice/crio-8b7638d1caa7aa7369b2923591331b5fc31cdd7de8796a13a99b1138840768a2 WatchSource:0}: Error finding container 8b7638d1caa7aa7369b2923591331b5fc31cdd7de8796a13a99b1138840768a2: Status 404 returned error can't find the container with id 8b7638d1caa7aa7369b2923591331b5fc31cdd7de8796a13a99b1138840768a2
Oct 30 19:27:37.817821 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:37.814855    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f15f13-22hc6" event={"ID":"888f7b8b-5ffa-40b6-b428-ae77ee29c855","Type":"ContainerStarted","Data":"f0ad75a101159fbb67097c30fb0703575d1c8342e584b058ac6486a0128811fa"}
Oct 30 19:27:37.817821 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:37.814936    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f15f13-22hc6" event={"ID":"888f7b8b-5ffa-40b6-b428-ae77ee29c855","Type":"ContainerStarted","Data":"8b7638d1caa7aa7369b2923591331b5fc31cdd7de8796a13a99b1138840768a2"}
Oct 30 19:27:37.833301 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:37.833222    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f15f13-22hc6" podStartSLOduration=1.83320254 podStartE2EDuration="1.83320254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:37.832260461 +0000 UTC m=+677.343320425" watchObservedRunningTime="2026-10-30 19:27:37.83320254 +0000 UTC m=+677.344262505"
Oct 30 19:27:38.069628 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.067795    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-921f1fba-wqgxb"
Oct 30 19:27:38.115925 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.115896    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e57ba6-c6bq7"
Oct 30 19:27:38.174885 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.174856    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58817e5e-c488-42bb-8de7-76df0d2652fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58817e5e-c488-42bb-8de7-76df0d2652fa-maas-api-openshift-ca\") pod \"58817e5e-c488-42bb-8de7-76df0d2652fa\" (UID: \"58817e5e-c488-42bb-8de7-76df0d2652fa\") "
Oct 30 19:27:38.175024 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.174931    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52d38b83-2cbe-4e80-adae-f837d0c9115b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52d38b83-2cbe-4e80-adae-f837d0c9115b-maas-api-xks-ca\") pod \"52d38b83-2cbe-4e80-adae-f837d0c9115b\" (UID: \"52d38b83-2cbe-4e80-adae-f837d0c9115b\") "
Oct 30 19:27:38.175024 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.174972    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52d38b83-2cbe-4e80-adae-f837d0c9115b-kube-api-access-sgzvr\" (UniqueName: \"kubernetes.io/projected/52d38b83-2cbe-4e80-adae-f837d0c9115b-kube-api-access-sgzvr\") pod \"52d38b83-2cbe-4e80-adae-f837d0c9115b\" (UID: \"52d38b83-2cbe-4e80-adae-f837d0c9115b\") "
Oct 30 19:27:38.175199 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.175052    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52d38b83-2cbe-4e80-adae-f837d0c9115b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52d38b83-2cbe-4e80-adae-f837d0c9115b-maas-api-openshift-ca\") pod \"52d38b83-2cbe-4e80-adae-f837d0c9115b\" (UID: \"52d38b83-2cbe-4e80-adae-f837d0c9115b\") "
Oct 30 19:27:38.175263 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.175199    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58817e5e-c488-42bb-8de7-76df0d2652fa-kube-api-access-xthz2\" (UniqueName: \"kubernetes.io/projected/58817e5e-c488-42bb-8de7-76df0d2652fa-kube-api-access-xthz2\") pod \"58817e5e-c488-42bb-8de7-76df0d2652fa\" (UID: \"58817e5e-c488-42bb-8de7-76df0d2652fa\") "
Oct 30 19:27:38.175263 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.175244    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58817e5e-c488-42bb-8de7-76df0d2652fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58817e5e-c488-42bb-8de7-76df0d2652fa-maas-api-xks-ca\") pod \"58817e5e-c488-42bb-8de7-76df0d2652fa\" (UID: \"58817e5e-c488-42bb-8de7-76df0d2652fa\") "
Oct 30 19:27:38.175367 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.175247    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58817e5e-c488-42bb-8de7-76df0d2652fa-maas-api-openshift-ca" pod "58817e5e-c488-42bb-8de7-76df0d2652fa" (UID: "58817e5e-c488-42bb-8de7-76df0d2652fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:27:38.175708 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.175687    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58817e5e-c488-42bb-8de7-76df0d2652fa-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:38.177601 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.177572    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52d38b83-2cbe-4e80-adae-f837d0c9115b-maas-api-openshift-ca" pod "52d38b83-2cbe-4e80-adae-f837d0c9115b" (UID: "52d38b83-2cbe-4e80-adae-f837d0c9115b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:27:38.178330 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.178287    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52d38b83-2cbe-4e80-adae-f837d0c9115b-maas-api-xks-ca" pod "52d38b83-2cbe-4e80-adae-f837d0c9115b" (UID: "52d38b83-2cbe-4e80-adae-f837d0c9115b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:27:38.178452 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.178430    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58817e5e-c488-42bb-8de7-76df0d2652fa-maas-api-xks-ca" pod "58817e5e-c488-42bb-8de7-76df0d2652fa" (UID: "58817e5e-c488-42bb-8de7-76df0d2652fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:27:38.182076 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.182049    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52d38b83-2cbe-4e80-adae-f837d0c9115b-kube-api-access-sgzvr" pod "52d38b83-2cbe-4e80-adae-f837d0c9115b" (UID: "52d38b83-2cbe-4e80-adae-f837d0c9115b"). InnerVolumeSpecName "kube-api-access-sgzvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:27:38.185431 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.185395    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58817e5e-c488-42bb-8de7-76df0d2652fa-kube-api-access-xthz2" pod "58817e5e-c488-42bb-8de7-76df0d2652fa" (UID: "58817e5e-c488-42bb-8de7-76df0d2652fa"). InnerVolumeSpecName "kube-api-access-xthz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:27:38.276873 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.276831    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52d38b83-2cbe-4e80-adae-f837d0c9115b-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:38.276873 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.276872    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xthz2\" (UniqueName: \"kubernetes.io/projected/58817e5e-c488-42bb-8de7-76df0d2652fa-kube-api-access-xthz2\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:38.277095 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.276886    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58817e5e-c488-42bb-8de7-76df0d2652fa-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:38.277095 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.276897    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52d38b83-2cbe-4e80-adae-f837d0c9115b-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:38.277095 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.276907    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgzvr\" (UniqueName: \"kubernetes.io/projected/52d38b83-2cbe-4e80-adae-f837d0c9115b-kube-api-access-sgzvr\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:38.311954 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.311918    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9832672-dzt6l"]
Oct 30 19:27:38.312545 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.312529    5043 cpu_manager.go:414] "containerMap: removing container" podUID="58817e5e-c488-42bb-8de7-76df0d2652fa" containerName="revoke-keys"
Oct 30 19:27:38.312603 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.312547    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58817e5e-c488-42bb-8de7-76df0d2652fa" containerName="revoke-keys"
Oct 30 19:27:38.312603 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.312557    5043 cpu_manager.go:414] "containerMap: removing container" podUID="52d38b83-2cbe-4e80-adae-f837d0c9115b" containerName="revoke-keys"
Oct 30 19:27:38.312603 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.312562    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52d38b83-2cbe-4e80-adae-f837d0c9115b" containerName="revoke-keys"
Oct 30 19:27:38.312700 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.312655    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="58817e5e-c488-42bb-8de7-76df0d2652fa" containerName="revoke-keys"
Oct 30 19:27:38.312700 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.312667    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="52d38b83-2cbe-4e80-adae-f837d0c9115b" containerName="revoke-keys"
Oct 30 19:27:38.316060 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.316039    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9832672-dzt6l"
Oct 30 19:27:38.325633 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.325561    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9832672-dzt6l"]
Oct 30 19:27:38.364457 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.364424    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9565d20-7wl65"]
Oct 30 19:27:38.369811 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.369704    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9565d20-7wl65"
Oct 30 19:27:38.377474 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.377445    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9565d20-7wl65"]
Oct 30 19:27:38.378109 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.378089    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5c72961-9b6a-4ae1-a089-03d8f2824672-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9832672-dzt6l\" (UID: \"b5c72961-9b6a-4ae1-a089-03d8f2824672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9832672-dzt6l"
Oct 30 19:27:38.378232 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.378131    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5c72961-9b6a-4ae1-a089-03d8f2824672-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9832672-dzt6l\" (UID: \"b5c72961-9b6a-4ae1-a089-03d8f2824672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9832672-dzt6l"
Oct 30 19:27:38.378312 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.378284    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e2e5350-a94f-41bb-bea3-8fcd882b2552-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9565d20-7wl65\" (UID: \"6e2e5350-a94f-41bb-bea3-8fcd882b2552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9565d20-7wl65"
Oct 30 19:27:38.378480 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.378405    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjc9b\" (UniqueName: \"kubernetes.io/projected/6e2e5350-a94f-41bb-bea3-8fcd882b2552-kube-api-access-wjc9b\") pod \"maas-api-revoke-sub-d9565d20-7wl65\" (UID: \"6e2e5350-a94f-41bb-bea3-8fcd882b2552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9565d20-7wl65"
Oct 30 19:27:38.378722 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.378685    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vffm\" (UniqueName: \"kubernetes.io/projected/b5c72961-9b6a-4ae1-a089-03d8f2824672-kube-api-access-4vffm\") pod \"maas-api-revoke-sub-b9832672-dzt6l\" (UID: \"b5c72961-9b6a-4ae1-a089-03d8f2824672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9832672-dzt6l"
Oct 30 19:27:38.378884 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.378772    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e2e5350-a94f-41bb-bea3-8fcd882b2552-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9565d20-7wl65\" (UID: \"6e2e5350-a94f-41bb-bea3-8fcd882b2552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9565d20-7wl65"
Oct 30 19:27:38.479633 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.479591    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e2e5350-a94f-41bb-bea3-8fcd882b2552-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9565d20-7wl65\" (UID: \"6e2e5350-a94f-41bb-bea3-8fcd882b2552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9565d20-7wl65"
Oct 30 19:27:38.479819 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.479671    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjc9b\" (UniqueName: \"kubernetes.io/projected/6e2e5350-a94f-41bb-bea3-8fcd882b2552-kube-api-access-wjc9b\") pod \"maas-api-revoke-sub-d9565d20-7wl65\" (UID: \"6e2e5350-a94f-41bb-bea3-8fcd882b2552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9565d20-7wl65"
Oct 30 19:27:38.479819 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.479708    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vffm\" (UniqueName: \"kubernetes.io/projected/b5c72961-9b6a-4ae1-a089-03d8f2824672-kube-api-access-4vffm\") pod \"maas-api-revoke-sub-b9832672-dzt6l\" (UID: \"b5c72961-9b6a-4ae1-a089-03d8f2824672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9832672-dzt6l"
Oct 30 19:27:38.479819 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.479777    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e2e5350-a94f-41bb-bea3-8fcd882b2552-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9565d20-7wl65\" (UID: \"6e2e5350-a94f-41bb-bea3-8fcd882b2552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9565d20-7wl65"
Oct 30 19:27:38.479970 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.479902    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5c72961-9b6a-4ae1-a089-03d8f2824672-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9832672-dzt6l\" (UID: \"b5c72961-9b6a-4ae1-a089-03d8f2824672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9832672-dzt6l"
Oct 30 19:27:38.479970 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.479930    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5c72961-9b6a-4ae1-a089-03d8f2824672-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9832672-dzt6l\" (UID: \"b5c72961-9b6a-4ae1-a089-03d8f2824672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9832672-dzt6l"
Oct 30 19:27:38.480575 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.480547    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e2e5350-a94f-41bb-bea3-8fcd882b2552-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9565d20-7wl65\" (UID: \"6e2e5350-a94f-41bb-bea3-8fcd882b2552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9565d20-7wl65"
Oct 30 19:27:38.480773 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.480727    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5c72961-9b6a-4ae1-a089-03d8f2824672-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9832672-dzt6l\" (UID: \"b5c72961-9b6a-4ae1-a089-03d8f2824672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9832672-dzt6l"
Oct 30 19:27:38.482298 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.482281    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5c72961-9b6a-4ae1-a089-03d8f2824672-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9832672-dzt6l\" (UID: \"b5c72961-9b6a-4ae1-a089-03d8f2824672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9832672-dzt6l"
Oct 30 19:27:38.482532 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.482517    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e2e5350-a94f-41bb-bea3-8fcd882b2552-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9565d20-7wl65\" (UID: \"6e2e5350-a94f-41bb-bea3-8fcd882b2552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9565d20-7wl65"
Oct 30 19:27:38.487911 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.487885    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vffm\" (UniqueName: \"kubernetes.io/projected/b5c72961-9b6a-4ae1-a089-03d8f2824672-kube-api-access-4vffm\") pod \"maas-api-revoke-sub-b9832672-dzt6l\" (UID: \"b5c72961-9b6a-4ae1-a089-03d8f2824672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9832672-dzt6l"
Oct 30 19:27:38.488007 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.487977    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjc9b\" (UniqueName: \"kubernetes.io/projected/6e2e5350-a94f-41bb-bea3-8fcd882b2552-kube-api-access-wjc9b\") pod \"maas-api-revoke-sub-d9565d20-7wl65\" (UID: \"6e2e5350-a94f-41bb-bea3-8fcd882b2552\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9565d20-7wl65"
Oct 30 19:27:38.635974 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.635889    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9832672-dzt6l"
Oct 30 19:27:38.690673 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.690627    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9565d20-7wl65"
Oct 30 19:27:38.823875 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.823307    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e57ba6-c6bq7"
Oct 30 19:27:38.823875 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.823327    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e57ba6-c6bq7" event={"ID":"52d38b83-2cbe-4e80-adae-f837d0c9115b","Type":"ContainerDied","Data":"b7dc3afc9784330cd769d5f31c0052ae0c9c69bef2af6a0b5aacf2d3b21c7eb0"}
Oct 30 19:27:38.823875 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.823372    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7dc3afc9784330cd769d5f31c0052ae0c9c69bef2af6a0b5aacf2d3b21c7eb0"
Oct 30 19:27:38.825527 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.825501    5043 generic.go:356] "Generic (PLEG): container finished" podID="888f7b8b-5ffa-40b6-b428-ae77ee29c855" containerID="f0ad75a101159fbb67097c30fb0703575d1c8342e584b058ac6486a0128811fa" exitCode=0
Oct 30 19:27:38.825644 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.825596    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f15f13-22hc6" event={"ID":"888f7b8b-5ffa-40b6-b428-ae77ee29c855","Type":"ContainerDied","Data":"f0ad75a101159fbb67097c30fb0703575d1c8342e584b058ac6486a0128811fa"}
Oct 30 19:27:38.827192 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.827172    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-921f1fba-wqgxb" event={"ID":"58817e5e-c488-42bb-8de7-76df0d2652fa","Type":"ContainerDied","Data":"0b655d227a4061e7f248aa6b541c77b68f62682ad28b9cd2562740ee4fbe00cf"}
Oct 30 19:27:38.827272 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.827199    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-921f1fba-wqgxb"
Oct 30 19:27:38.827272 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.827209    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b655d227a4061e7f248aa6b541c77b68f62682ad28b9cd2562740ee4fbe00cf"
Oct 30 19:27:38.923834 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:27:38.923799    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5c72961_9b6a_4ae1_a089_03d8f2824672.slice/crio-23615c85723ab797e99433e4edd8f761f64814e84bd0b80bcdb805463c110e74 WatchSource:0}: Error finding container 23615c85723ab797e99433e4edd8f761f64814e84bd0b80bcdb805463c110e74: Status 404 returned error can't find the container with id 23615c85723ab797e99433e4edd8f761f64814e84bd0b80bcdb805463c110e74
Oct 30 19:27:38.925133 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.925108    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9832672-dzt6l"]
Oct 30 19:27:38.997112 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:38.997083    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9565d20-7wl65"]
Oct 30 19:27:39.059101 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:27:39.059067    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e2e5350_a94f_41bb_bea3_8fcd882b2552.slice/crio-3d0900ef38d14e3d22e34a7c9d8af6842a2eccf5a1669399b74fc7eee196d93f WatchSource:0}: Error finding container 3d0900ef38d14e3d22e34a7c9d8af6842a2eccf5a1669399b74fc7eee196d93f: Status 404 returned error can't find the container with id 3d0900ef38d14e3d22e34a7c9d8af6842a2eccf5a1669399b74fc7eee196d93f
Oct 30 19:27:39.835200 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:39.835165    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9565d20-7wl65" event={"ID":"6e2e5350-a94f-41bb-bea3-8fcd882b2552","Type":"ContainerStarted","Data":"180b850d30e20cf38e85da4c96a355d45fc4b5fa2ac0afef708bf85b376f7f2d"}
Oct 30 19:27:39.835673 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:39.835216    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9565d20-7wl65" event={"ID":"6e2e5350-a94f-41bb-bea3-8fcd882b2552","Type":"ContainerStarted","Data":"3d0900ef38d14e3d22e34a7c9d8af6842a2eccf5a1669399b74fc7eee196d93f"}
Oct 30 19:27:39.836861 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:39.836839    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9832672-dzt6l" event={"ID":"b5c72961-9b6a-4ae1-a089-03d8f2824672","Type":"ContainerStarted","Data":"0df51b9498fd8afead5fe3f409db6fe5a704ad1c23fd728f6b8a8743c7125ebd"}
Oct 30 19:27:39.836973 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:39.836885    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9832672-dzt6l" event={"ID":"b5c72961-9b6a-4ae1-a089-03d8f2824672","Type":"ContainerStarted","Data":"23615c85723ab797e99433e4edd8f761f64814e84bd0b80bcdb805463c110e74"}
Oct 30 19:27:39.855835 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:39.855792    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9565d20-7wl65" podStartSLOduration=1.855778308 podStartE2EDuration="1.855778308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:39.853006476 +0000 UTC m=+679.364066452" watchObservedRunningTime="2026-10-30 19:27:39.855778308 +0000 UTC m=+679.366838273"
Oct 30 19:27:39.873878 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:39.873822    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9832672-dzt6l" podStartSLOduration=1.873805093 podStartE2EDuration="1.873805093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:39.870014787 +0000 UTC m=+679.381074752" watchObservedRunningTime="2026-10-30 19:27:39.873805093 +0000 UTC m=+679.384865060"
Oct 30 19:27:39.972585 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:39.972549    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61156d8c-x9bk8"]
Oct 30 19:27:39.977128 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:39.977107    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61156d8c-x9bk8"
Oct 30 19:27:39.999845 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:39.999813    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61156d8c-x9bk8"]
Oct 30 19:27:40.002528 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:40.002499    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wgmd\" (UniqueName: \"kubernetes.io/projected/789d2679-ca00-4aeb-850a-a81e96ee7bb5-kube-api-access-6wgmd\") pod \"maas-api-revoke-sub-61156d8c-x9bk8\" (UID: \"789d2679-ca00-4aeb-850a-a81e96ee7bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61156d8c-x9bk8"
Oct 30 19:27:40.002705 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:40.002686    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/789d2679-ca00-4aeb-850a-a81e96ee7bb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61156d8c-x9bk8\" (UID: \"789d2679-ca00-4aeb-850a-a81e96ee7bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61156d8c-x9bk8"
Oct 30 19:27:40.002963 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:40.002923    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/789d2679-ca00-4aeb-850a-a81e96ee7bb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61156d8c-x9bk8\" (UID: \"789d2679-ca00-4aeb-850a-a81e96ee7bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61156d8c-x9bk8"
Oct 30 19:27:40.042621 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:40.042595    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f15f13-22hc6"
Oct 30 19:27:40.104417 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:40.104350    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/888f7b8b-5ffa-40b6-b428-ae77ee29c855-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/888f7b8b-5ffa-40b6-b428-ae77ee29c855-maas-api-openshift-ca\") pod \"888f7b8b-5ffa-40b6-b428-ae77ee29c855\" (UID: \"888f7b8b-5ffa-40b6-b428-ae77ee29c855\") "
Oct 30 19:27:40.104549 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:40.104479    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/888f7b8b-5ffa-40b6-b428-ae77ee29c855-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/888f7b8b-5ffa-40b6-b428-ae77ee29c855-maas-api-xks-ca\") pod \"888f7b8b-5ffa-40b6-b428-ae77ee29c855\" (UID: \"888f7b8b-5ffa-40b6-b428-ae77ee29c855\") "
Oct 30 19:27:40.104549 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:40.104505    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/888f7b8b-5ffa-40b6-b428-ae77ee29c855-kube-api-access-kpdhx\" (UniqueName: \"kubernetes.io/projected/888f7b8b-5ffa-40b6-b428-ae77ee29c855-kube-api-access-kpdhx\") pod \"888f7b8b-5ffa-40b6-b428-ae77ee29c855\" (UID: \"888f7b8b-5ffa-40b6-b428-ae77ee29c855\") "
Oct 30 19:27:40.104704 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:40.104685    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/789d2679-ca00-4aeb-850a-a81e96ee7bb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61156d8c-x9bk8\" (UID: \"789d2679-ca00-4aeb-850a-a81e96ee7bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61156d8c-x9bk8"
Oct 30 19:27:40.104801 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:40.104770    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/888f7b8b-5ffa-40b6-b428-ae77ee29c855-maas-api-openshift-ca" pod "888f7b8b-5ffa-40b6-b428-ae77ee29c855" (UID: "888f7b8b-5ffa-40b6-b428-ae77ee29c855"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:27:40.104921 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:40.104902    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wgmd\" (UniqueName: \"kubernetes.io/projected/789d2679-ca00-4aeb-850a-a81e96ee7bb5-kube-api-access-6wgmd\") pod \"maas-api-revoke-sub-61156d8c-x9bk8\" (UID: \"789d2679-ca00-4aeb-850a-a81e96ee7bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61156d8c-x9bk8"
Oct 30 19:27:40.104978 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:40.104955    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/789d2679-ca00-4aeb-850a-a81e96ee7bb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61156d8c-x9bk8\" (UID: \"789d2679-ca00-4aeb-850a-a81e96ee7bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61156d8c-x9bk8"
Oct 30 19:27:40.105433 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:40.105412    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/888f7b8b-5ffa-40b6-b428-ae77ee29c855-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:40.105525 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:40.105441    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/789d2679-ca00-4aeb-850a-a81e96ee7bb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61156d8c-x9bk8\" (UID: \"789d2679-ca00-4aeb-850a-a81e96ee7bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61156d8c-x9bk8"
Oct 30 19:27:40.107217 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:40.107191    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/888f7b8b-5ffa-40b6-b428-ae77ee29c855-maas-api-xks-ca" pod "888f7b8b-5ffa-40b6-b428-ae77ee29c855" (UID: "888f7b8b-5ffa-40b6-b428-ae77ee29c855"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:27:40.107660 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:40.107638    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/789d2679-ca00-4aeb-850a-a81e96ee7bb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61156d8c-x9bk8\" (UID: \"789d2679-ca00-4aeb-850a-a81e96ee7bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61156d8c-x9bk8"
Oct 30 19:27:40.107999 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:40.107981    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/888f7b8b-5ffa-40b6-b428-ae77ee29c855-kube-api-access-kpdhx" pod "888f7b8b-5ffa-40b6-b428-ae77ee29c855" (UID: "888f7b8b-5ffa-40b6-b428-ae77ee29c855"). InnerVolumeSpecName "kube-api-access-kpdhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:27:40.113712 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:40.113689    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wgmd\" (UniqueName: \"kubernetes.io/projected/789d2679-ca00-4aeb-850a-a81e96ee7bb5-kube-api-access-6wgmd\") pod \"maas-api-revoke-sub-61156d8c-x9bk8\" (UID: \"789d2679-ca00-4aeb-850a-a81e96ee7bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61156d8c-x9bk8"
Oct 30 19:27:40.206885 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:40.206856    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/888f7b8b-5ffa-40b6-b428-ae77ee29c855-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:40.207001 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:40.206888    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpdhx\" (UniqueName: \"kubernetes.io/projected/888f7b8b-5ffa-40b6-b428-ae77ee29c855-kube-api-access-kpdhx\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:40.319789 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:40.319760    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61156d8c-x9bk8"
Oct 30 19:27:40.629694 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:40.629666    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61156d8c-x9bk8"]
Oct 30 19:27:40.630160 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:27:40.630137    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod789d2679_ca00_4aeb_850a_a81e96ee7bb5.slice/crio-a32203fbfd0f2118ef20b5218265e4acaaff8308ee88682e53174c54816bef40 WatchSource:0}: Error finding container a32203fbfd0f2118ef20b5218265e4acaaff8308ee88682e53174c54816bef40: Status 404 returned error can't find the container with id a32203fbfd0f2118ef20b5218265e4acaaff8308ee88682e53174c54816bef40
Oct 30 19:27:40.845366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:40.845339    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f15f13-22hc6"
Oct 30 19:27:40.845814 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:40.845333    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f15f13-22hc6" event={"ID":"888f7b8b-5ffa-40b6-b428-ae77ee29c855","Type":"ContainerDied","Data":"8b7638d1caa7aa7369b2923591331b5fc31cdd7de8796a13a99b1138840768a2"}
Oct 30 19:27:40.845814 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:40.845440    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b7638d1caa7aa7369b2923591331b5fc31cdd7de8796a13a99b1138840768a2"
Oct 30 19:27:40.847114 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:40.847086    5043 generic.go:356] "Generic (PLEG): container finished" podID="b5c72961-9b6a-4ae1-a089-03d8f2824672" containerID="0df51b9498fd8afead5fe3f409db6fe5a704ad1c23fd728f6b8a8743c7125ebd" exitCode=0
Oct 30 19:27:40.847228 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:40.847171    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9832672-dzt6l" event={"ID":"b5c72961-9b6a-4ae1-a089-03d8f2824672","Type":"ContainerDied","Data":"0df51b9498fd8afead5fe3f409db6fe5a704ad1c23fd728f6b8a8743c7125ebd"}
Oct 30 19:27:40.848704 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:40.848678    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61156d8c-x9bk8" event={"ID":"789d2679-ca00-4aeb-850a-a81e96ee7bb5","Type":"ContainerStarted","Data":"a32203fbfd0f2118ef20b5218265e4acaaff8308ee88682e53174c54816bef40"}
Oct 30 19:27:40.850210 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:40.850187    5043 generic.go:356] "Generic (PLEG): container finished" podID="6e2e5350-a94f-41bb-bea3-8fcd882b2552" containerID="180b850d30e20cf38e85da4c96a355d45fc4b5fa2ac0afef708bf85b376f7f2d" exitCode=0
Oct 30 19:27:40.850280 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:40.850229    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9565d20-7wl65" event={"ID":"6e2e5350-a94f-41bb-bea3-8fcd882b2552","Type":"ContainerDied","Data":"180b850d30e20cf38e85da4c96a355d45fc4b5fa2ac0afef708bf85b376f7f2d"}
Oct 30 19:27:41.861702 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:41.861597    5043 generic.go:356] "Generic (PLEG): container finished" podID="789d2679-ca00-4aeb-850a-a81e96ee7bb5" containerID="3aa0d45a656082438385ec881f85cd46a257b5ed09e89938b4cde963ace10a45" exitCode=0
Oct 30 19:27:41.862144 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:41.861791    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61156d8c-x9bk8" event={"ID":"789d2679-ca00-4aeb-850a-a81e96ee7bb5","Type":"ContainerDied","Data":"3aa0d45a656082438385ec881f85cd46a257b5ed09e89938b4cde963ace10a45"}
Oct 30 19:27:42.053118 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:42.053091    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9832672-dzt6l"
Oct 30 19:27:42.100223 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:42.100197    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9565d20-7wl65"
Oct 30 19:27:42.130682 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:42.130611    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5c72961-9b6a-4ae1-a089-03d8f2824672-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5c72961-9b6a-4ae1-a089-03d8f2824672-maas-api-xks-ca\") pod \"b5c72961-9b6a-4ae1-a089-03d8f2824672\" (UID: \"b5c72961-9b6a-4ae1-a089-03d8f2824672\") "
Oct 30 19:27:42.130878 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:42.130688    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5c72961-9b6a-4ae1-a089-03d8f2824672-kube-api-access-4vffm\" (UniqueName: \"kubernetes.io/projected/b5c72961-9b6a-4ae1-a089-03d8f2824672-kube-api-access-4vffm\") pod \"b5c72961-9b6a-4ae1-a089-03d8f2824672\" (UID: \"b5c72961-9b6a-4ae1-a089-03d8f2824672\") "
Oct 30 19:27:42.130878 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:42.130720    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e2e5350-a94f-41bb-bea3-8fcd882b2552-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e2e5350-a94f-41bb-bea3-8fcd882b2552-maas-api-openshift-ca\") pod \"6e2e5350-a94f-41bb-bea3-8fcd882b2552\" (UID: \"6e2e5350-a94f-41bb-bea3-8fcd882b2552\") "
Oct 30 19:27:42.130996 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:42.130885    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e2e5350-a94f-41bb-bea3-8fcd882b2552-kube-api-access-wjc9b\" (UniqueName: \"kubernetes.io/projected/6e2e5350-a94f-41bb-bea3-8fcd882b2552-kube-api-access-wjc9b\") pod \"6e2e5350-a94f-41bb-bea3-8fcd882b2552\" (UID: \"6e2e5350-a94f-41bb-bea3-8fcd882b2552\") "
Oct 30 19:27:42.130996 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:42.130920    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5c72961-9b6a-4ae1-a089-03d8f2824672-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5c72961-9b6a-4ae1-a089-03d8f2824672-maas-api-openshift-ca\") pod \"b5c72961-9b6a-4ae1-a089-03d8f2824672\" (UID: \"b5c72961-9b6a-4ae1-a089-03d8f2824672\") "
Oct 30 19:27:42.130996 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:42.130979    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e2e5350-a94f-41bb-bea3-8fcd882b2552-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e2e5350-a94f-41bb-bea3-8fcd882b2552-maas-api-xks-ca\") pod \"6e2e5350-a94f-41bb-bea3-8fcd882b2552\" (UID: \"6e2e5350-a94f-41bb-bea3-8fcd882b2552\") "
Oct 30 19:27:42.132022 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:42.131981    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e2e5350-a94f-41bb-bea3-8fcd882b2552-maas-api-openshift-ca" pod "6e2e5350-a94f-41bb-bea3-8fcd882b2552" (UID: "6e2e5350-a94f-41bb-bea3-8fcd882b2552"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:27:42.132580 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:42.132558    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5c72961-9b6a-4ae1-a089-03d8f2824672-maas-api-openshift-ca" pod "b5c72961-9b6a-4ae1-a089-03d8f2824672" (UID: "b5c72961-9b6a-4ae1-a089-03d8f2824672"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:27:42.134059 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:42.134032    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e2e5350-a94f-41bb-bea3-8fcd882b2552-maas-api-xks-ca" pod "6e2e5350-a94f-41bb-bea3-8fcd882b2552" (UID: "6e2e5350-a94f-41bb-bea3-8fcd882b2552"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:27:42.134188 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:42.134160    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5c72961-9b6a-4ae1-a089-03d8f2824672-maas-api-xks-ca" pod "b5c72961-9b6a-4ae1-a089-03d8f2824672" (UID: "b5c72961-9b6a-4ae1-a089-03d8f2824672"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:27:42.135167 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:42.135142    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5c72961-9b6a-4ae1-a089-03d8f2824672-kube-api-access-4vffm" pod "b5c72961-9b6a-4ae1-a089-03d8f2824672" (UID: "b5c72961-9b6a-4ae1-a089-03d8f2824672"). InnerVolumeSpecName "kube-api-access-4vffm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:27:42.135236 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:42.135175    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e2e5350-a94f-41bb-bea3-8fcd882b2552-kube-api-access-wjc9b" pod "6e2e5350-a94f-41bb-bea3-8fcd882b2552" (UID: "6e2e5350-a94f-41bb-bea3-8fcd882b2552"). InnerVolumeSpecName "kube-api-access-wjc9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:27:42.232405 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:42.232369    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e2e5350-a94f-41bb-bea3-8fcd882b2552-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:42.232405 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:42.232401    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5c72961-9b6a-4ae1-a089-03d8f2824672-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:42.232405 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:42.232411    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vffm\" (UniqueName: \"kubernetes.io/projected/b5c72961-9b6a-4ae1-a089-03d8f2824672-kube-api-access-4vffm\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:42.232635 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:42.232423    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e2e5350-a94f-41bb-bea3-8fcd882b2552-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:42.232635 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:42.232434    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjc9b\" (UniqueName: \"kubernetes.io/projected/6e2e5350-a94f-41bb-bea3-8fcd882b2552-kube-api-access-wjc9b\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:42.232635 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:42.232444    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5c72961-9b6a-4ae1-a089-03d8f2824672-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:42.869547 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:42.869516    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9565d20-7wl65"
Oct 30 19:27:42.869991 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:42.869542    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9565d20-7wl65" event={"ID":"6e2e5350-a94f-41bb-bea3-8fcd882b2552","Type":"ContainerDied","Data":"3d0900ef38d14e3d22e34a7c9d8af6842a2eccf5a1669399b74fc7eee196d93f"}
Oct 30 19:27:42.869991 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:42.869598    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d0900ef38d14e3d22e34a7c9d8af6842a2eccf5a1669399b74fc7eee196d93f"
Oct 30 19:27:42.871552 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:42.871533    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9832672-dzt6l"
Oct 30 19:27:42.871673 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:42.871547    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9832672-dzt6l" event={"ID":"b5c72961-9b6a-4ae1-a089-03d8f2824672","Type":"ContainerDied","Data":"23615c85723ab797e99433e4edd8f761f64814e84bd0b80bcdb805463c110e74"}
Oct 30 19:27:42.871673 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:42.871590    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23615c85723ab797e99433e4edd8f761f64814e84bd0b80bcdb805463c110e74"
Oct 30 19:27:43.001433 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:43.001396    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61156d8c-x9bk8"
Oct 30 19:27:43.040789 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:43.040714    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/789d2679-ca00-4aeb-850a-a81e96ee7bb5-kube-api-access-6wgmd\" (UniqueName: \"kubernetes.io/projected/789d2679-ca00-4aeb-850a-a81e96ee7bb5-kube-api-access-6wgmd\") pod \"789d2679-ca00-4aeb-850a-a81e96ee7bb5\" (UID: \"789d2679-ca00-4aeb-850a-a81e96ee7bb5\") "
Oct 30 19:27:43.040973 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:43.040846    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/789d2679-ca00-4aeb-850a-a81e96ee7bb5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/789d2679-ca00-4aeb-850a-a81e96ee7bb5-maas-api-xks-ca\") pod \"789d2679-ca00-4aeb-850a-a81e96ee7bb5\" (UID: \"789d2679-ca00-4aeb-850a-a81e96ee7bb5\") "
Oct 30 19:27:43.041041 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:43.041008    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/789d2679-ca00-4aeb-850a-a81e96ee7bb5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/789d2679-ca00-4aeb-850a-a81e96ee7bb5-maas-api-openshift-ca\") pod \"789d2679-ca00-4aeb-850a-a81e96ee7bb5\" (UID: \"789d2679-ca00-4aeb-850a-a81e96ee7bb5\") "
Oct 30 19:27:43.041348 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:43.041321    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/789d2679-ca00-4aeb-850a-a81e96ee7bb5-maas-api-openshift-ca" pod "789d2679-ca00-4aeb-850a-a81e96ee7bb5" (UID: "789d2679-ca00-4aeb-850a-a81e96ee7bb5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:27:43.041651 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:43.041449    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/789d2679-ca00-4aeb-850a-a81e96ee7bb5-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:43.043110 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:43.043081    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/789d2679-ca00-4aeb-850a-a81e96ee7bb5-maas-api-xks-ca" pod "789d2679-ca00-4aeb-850a-a81e96ee7bb5" (UID: "789d2679-ca00-4aeb-850a-a81e96ee7bb5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:27:43.043448 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:43.043430    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/789d2679-ca00-4aeb-850a-a81e96ee7bb5-kube-api-access-6wgmd" pod "789d2679-ca00-4aeb-850a-a81e96ee7bb5" (UID: "789d2679-ca00-4aeb-850a-a81e96ee7bb5"). InnerVolumeSpecName "kube-api-access-6wgmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:27:43.142136 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:43.142106    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/789d2679-ca00-4aeb-850a-a81e96ee7bb5-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:43.142136 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:43.142134    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wgmd\" (UniqueName: \"kubernetes.io/projected/789d2679-ca00-4aeb-850a-a81e96ee7bb5-kube-api-access-6wgmd\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:43.888161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:43.888135    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61156d8c-x9bk8"
Oct 30 19:27:43.888637 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:43.888186    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61156d8c-x9bk8" event={"ID":"789d2679-ca00-4aeb-850a-a81e96ee7bb5","Type":"ContainerDied","Data":"a32203fbfd0f2118ef20b5218265e4acaaff8308ee88682e53174c54816bef40"}
Oct 30 19:27:43.888637 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:43.888237    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a32203fbfd0f2118ef20b5218265e4acaaff8308ee88682e53174c54816bef40"
Oct 30 19:27:47.757566 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:47.757519    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84d1285b-drd4t"]
Oct 30 19:27:47.758102 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:47.758065    5043 cpu_manager.go:414] "containerMap: removing container" podUID="6e2e5350-a94f-41bb-bea3-8fcd882b2552" containerName="revoke-keys"
Oct 30 19:27:47.758102 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:47.758078    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e2e5350-a94f-41bb-bea3-8fcd882b2552" containerName="revoke-keys"
Oct 30 19:27:47.758102 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:47.758101    5043 cpu_manager.go:414] "containerMap: removing container" podUID="b5c72961-9b6a-4ae1-a089-03d8f2824672" containerName="revoke-keys"
Oct 30 19:27:47.758255 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:47.758106    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5c72961-9b6a-4ae1-a089-03d8f2824672" containerName="revoke-keys"
Oct 30 19:27:47.758255 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:47.758123    5043 cpu_manager.go:414] "containerMap: removing container" podUID="789d2679-ca00-4aeb-850a-a81e96ee7bb5" containerName="revoke-keys"
Oct 30 19:27:47.758255 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:47.758128    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="789d2679-ca00-4aeb-850a-a81e96ee7bb5" containerName="revoke-keys"
Oct 30 19:27:47.758255 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:47.758148    5043 cpu_manager.go:414] "containerMap: removing container" podUID="888f7b8b-5ffa-40b6-b428-ae77ee29c855" containerName="revoke-keys"
Oct 30 19:27:47.758255 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:47.758152    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="888f7b8b-5ffa-40b6-b428-ae77ee29c855" containerName="revoke-keys"
Oct 30 19:27:47.758255 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:47.758220    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5c72961-9b6a-4ae1-a089-03d8f2824672" containerName="revoke-keys"
Oct 30 19:27:47.758255 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:47.758230    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="789d2679-ca00-4aeb-850a-a81e96ee7bb5" containerName="revoke-keys"
Oct 30 19:27:47.758255 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:47.758240    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="888f7b8b-5ffa-40b6-b428-ae77ee29c855" containerName="revoke-keys"
Oct 30 19:27:47.758255 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:47.758246    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e2e5350-a94f-41bb-bea3-8fcd882b2552" containerName="revoke-keys"
Oct 30 19:27:47.761909 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:47.761889    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d1285b-drd4t"
Oct 30 19:27:47.771216 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:47.771189    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:27:47.771397 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:47.771283    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:27:47.771397 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:47.771362    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:27:47.776356 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:47.774678    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:27:47.776356 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:47.776168    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84d1285b-drd4t"]
Oct 30 19:27:47.807392 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:47.807356    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnvbp\" (UniqueName: \"kubernetes.io/projected/dec8f742-4e6e-4f2d-819f-a30304039c5b-kube-api-access-dnvbp\") pod \"maas-api-revoke-sub-84d1285b-drd4t\" (UID: \"dec8f742-4e6e-4f2d-819f-a30304039c5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d1285b-drd4t"
Oct 30 19:27:47.807537 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:47.807438    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dec8f742-4e6e-4f2d-819f-a30304039c5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84d1285b-drd4t\" (UID: \"dec8f742-4e6e-4f2d-819f-a30304039c5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d1285b-drd4t"
Oct 30 19:27:47.807677 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:47.807650    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dec8f742-4e6e-4f2d-819f-a30304039c5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84d1285b-drd4t\" (UID: \"dec8f742-4e6e-4f2d-819f-a30304039c5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d1285b-drd4t"
Oct 30 19:27:47.909199 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:47.909166    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dec8f742-4e6e-4f2d-819f-a30304039c5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84d1285b-drd4t\" (UID: \"dec8f742-4e6e-4f2d-819f-a30304039c5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d1285b-drd4t"
Oct 30 19:27:47.909345 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:47.909278    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dec8f742-4e6e-4f2d-819f-a30304039c5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84d1285b-drd4t\" (UID: \"dec8f742-4e6e-4f2d-819f-a30304039c5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d1285b-drd4t"
Oct 30 19:27:47.909345 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:47.909327    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnvbp\" (UniqueName: \"kubernetes.io/projected/dec8f742-4e6e-4f2d-819f-a30304039c5b-kube-api-access-dnvbp\") pod \"maas-api-revoke-sub-84d1285b-drd4t\" (UID: \"dec8f742-4e6e-4f2d-819f-a30304039c5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d1285b-drd4t"
Oct 30 19:27:47.909905 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:47.909883    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dec8f742-4e6e-4f2d-819f-a30304039c5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84d1285b-drd4t\" (UID: \"dec8f742-4e6e-4f2d-819f-a30304039c5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d1285b-drd4t"
Oct 30 19:27:47.911816 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:47.911792    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dec8f742-4e6e-4f2d-819f-a30304039c5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84d1285b-drd4t\" (UID: \"dec8f742-4e6e-4f2d-819f-a30304039c5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d1285b-drd4t"
Oct 30 19:27:47.918102 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:47.918077    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnvbp\" (UniqueName: \"kubernetes.io/projected/dec8f742-4e6e-4f2d-819f-a30304039c5b-kube-api-access-dnvbp\") pod \"maas-api-revoke-sub-84d1285b-drd4t\" (UID: \"dec8f742-4e6e-4f2d-819f-a30304039c5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d1285b-drd4t"
Oct 30 19:27:48.078996 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:48.078915    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d1285b-drd4t"
Oct 30 19:27:48.378650 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:48.378618    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84d1285b-drd4t"]
Oct 30 19:27:48.379909 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:27:48.379880    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddec8f742_4e6e_4f2d_819f_a30304039c5b.slice/crio-aa8b5a609696d4919563577eb5976aa2998faae3e58b454c8edeba7dfcc0195e WatchSource:0}: Error finding container aa8b5a609696d4919563577eb5976aa2998faae3e58b454c8edeba7dfcc0195e: Status 404 returned error can't find the container with id aa8b5a609696d4919563577eb5976aa2998faae3e58b454c8edeba7dfcc0195e
Oct 30 19:27:48.469385 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:48.469354    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e16f494-qrpqh"]
Oct 30 19:27:48.474505 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:48.474483    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e16f494-qrpqh"
Oct 30 19:27:48.481184 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:48.481018    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e16f494-qrpqh"]
Oct 30 19:27:48.518623 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:48.518590    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93c44a79-bd6c-4c94-a74f-2cbf6d9240c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e16f494-qrpqh\" (UID: \"93c44a79-bd6c-4c94-a74f-2cbf6d9240c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e16f494-qrpqh"
Oct 30 19:27:48.518840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:48.518822    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93c44a79-bd6c-4c94-a74f-2cbf6d9240c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e16f494-qrpqh\" (UID: \"93c44a79-bd6c-4c94-a74f-2cbf6d9240c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e16f494-qrpqh"
Oct 30 19:27:48.518913 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:48.518858    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdr6q\" (UniqueName: \"kubernetes.io/projected/93c44a79-bd6c-4c94-a74f-2cbf6d9240c0-kube-api-access-xdr6q\") pod \"maas-api-revoke-sub-5e16f494-qrpqh\" (UID: \"93c44a79-bd6c-4c94-a74f-2cbf6d9240c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e16f494-qrpqh"
Oct 30 19:27:48.620208 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:48.620175    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93c44a79-bd6c-4c94-a74f-2cbf6d9240c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e16f494-qrpqh\" (UID: \"93c44a79-bd6c-4c94-a74f-2cbf6d9240c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e16f494-qrpqh"
Oct 30 19:27:48.620419 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:48.620294    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93c44a79-bd6c-4c94-a74f-2cbf6d9240c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e16f494-qrpqh\" (UID: \"93c44a79-bd6c-4c94-a74f-2cbf6d9240c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e16f494-qrpqh"
Oct 30 19:27:48.620419 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:48.620318    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdr6q\" (UniqueName: \"kubernetes.io/projected/93c44a79-bd6c-4c94-a74f-2cbf6d9240c0-kube-api-access-xdr6q\") pod \"maas-api-revoke-sub-5e16f494-qrpqh\" (UID: \"93c44a79-bd6c-4c94-a74f-2cbf6d9240c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e16f494-qrpqh"
Oct 30 19:27:48.621174 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:48.621142    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93c44a79-bd6c-4c94-a74f-2cbf6d9240c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e16f494-qrpqh\" (UID: \"93c44a79-bd6c-4c94-a74f-2cbf6d9240c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e16f494-qrpqh"
Oct 30 19:27:48.622753 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:48.622711    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93c44a79-bd6c-4c94-a74f-2cbf6d9240c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e16f494-qrpqh\" (UID: \"93c44a79-bd6c-4c94-a74f-2cbf6d9240c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e16f494-qrpqh"
Oct 30 19:27:48.628810 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:48.628754    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdr6q\" (UniqueName: \"kubernetes.io/projected/93c44a79-bd6c-4c94-a74f-2cbf6d9240c0-kube-api-access-xdr6q\") pod \"maas-api-revoke-sub-5e16f494-qrpqh\" (UID: \"93c44a79-bd6c-4c94-a74f-2cbf6d9240c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e16f494-qrpqh"
Oct 30 19:27:48.793471 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:48.793436    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e16f494-qrpqh"
Oct 30 19:27:48.934432 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:48.934393    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d1285b-drd4t" event={"ID":"dec8f742-4e6e-4f2d-819f-a30304039c5b","Type":"ContainerStarted","Data":"3b8f9dd58831d2329512bc76b88e9582cd4b0f9447d21da7a60354d543f1483f"}
Oct 30 19:27:48.934602 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:48.934460    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d1285b-drd4t" event={"ID":"dec8f742-4e6e-4f2d-819f-a30304039c5b","Type":"ContainerStarted","Data":"aa8b5a609696d4919563577eb5976aa2998faae3e58b454c8edeba7dfcc0195e"}
Oct 30 19:27:48.954236 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:48.954173    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d1285b-drd4t" podStartSLOduration=1.95415424 podStartE2EDuration="1.95415424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:48.950129634 +0000 UTC m=+688.461189600" watchObservedRunningTime="2026-10-30 19:27:48.95415424 +0000 UTC m=+688.465214206"
Oct 30 19:27:49.082308 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:49.082280    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e16f494-qrpqh"]
Oct 30 19:27:49.083507 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:27:49.083481    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93c44a79_bd6c_4c94_a74f_2cbf6d9240c0.slice/crio-a5ed59b7cef412fc21b7a1cfe51c05de12a800cbc560d2d3439d5e9d8af9a73d WatchSource:0}: Error finding container a5ed59b7cef412fc21b7a1cfe51c05de12a800cbc560d2d3439d5e9d8af9a73d: Status 404 returned error can't find the container with id a5ed59b7cef412fc21b7a1cfe51c05de12a800cbc560d2d3439d5e9d8af9a73d
Oct 30 19:27:49.943043 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:49.942843    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e16f494-qrpqh" event={"ID":"93c44a79-bd6c-4c94-a74f-2cbf6d9240c0","Type":"ContainerStarted","Data":"9d33234436200af4ace587731a37c1290714fbc4785d4fdf4c1dda07ed4906c3"}
Oct 30 19:27:49.943043 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:49.942917    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e16f494-qrpqh" event={"ID":"93c44a79-bd6c-4c94-a74f-2cbf6d9240c0","Type":"ContainerStarted","Data":"a5ed59b7cef412fc21b7a1cfe51c05de12a800cbc560d2d3439d5e9d8af9a73d"}
Oct 30 19:27:49.944899 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:49.944867    5043 generic.go:356] "Generic (PLEG): container finished" podID="dec8f742-4e6e-4f2d-819f-a30304039c5b" containerID="3b8f9dd58831d2329512bc76b88e9582cd4b0f9447d21da7a60354d543f1483f" exitCode=0
Oct 30 19:27:49.945060 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:49.944910    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d1285b-drd4t" event={"ID":"dec8f742-4e6e-4f2d-819f-a30304039c5b","Type":"ContainerDied","Data":"3b8f9dd58831d2329512bc76b88e9582cd4b0f9447d21da7a60354d543f1483f"}
Oct 30 19:27:49.966526 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:49.966478    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e16f494-qrpqh" podStartSLOduration=1.966463839 podStartE2EDuration="1.966463839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:49.963149059 +0000 UTC m=+689.474209028" watchObservedRunningTime="2026-10-30 19:27:49.966463839 +0000 UTC m=+689.477523816"
Oct 30 19:27:50.681270 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:50.681230    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a476e910-t5dwj"]
Oct 30 19:27:50.686044 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:50.686021    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a476e910-t5dwj"
Oct 30 19:27:50.691060 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:50.691023    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a476e910-t5dwj"]
Oct 30 19:27:50.745104 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:50.745070    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f52531a-7cb1-416c-86c2-61db7f8128d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a476e910-t5dwj\" (UID: \"3f52531a-7cb1-416c-86c2-61db7f8128d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a476e910-t5dwj"
Oct 30 19:27:50.745249 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:50.745176    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f52531a-7cb1-416c-86c2-61db7f8128d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a476e910-t5dwj\" (UID: \"3f52531a-7cb1-416c-86c2-61db7f8128d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a476e910-t5dwj"
Oct 30 19:27:50.745249 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:50.745200    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bt8p\" (UniqueName: \"kubernetes.io/projected/3f52531a-7cb1-416c-86c2-61db7f8128d9-kube-api-access-9bt8p\") pod \"maas-api-revoke-sub-a476e910-t5dwj\" (UID: \"3f52531a-7cb1-416c-86c2-61db7f8128d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a476e910-t5dwj"
Oct 30 19:27:50.846042 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:50.846007    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f52531a-7cb1-416c-86c2-61db7f8128d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a476e910-t5dwj\" (UID: \"3f52531a-7cb1-416c-86c2-61db7f8128d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a476e910-t5dwj"
Oct 30 19:27:50.846227 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:50.846110    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f52531a-7cb1-416c-86c2-61db7f8128d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a476e910-t5dwj\" (UID: \"3f52531a-7cb1-416c-86c2-61db7f8128d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a476e910-t5dwj"
Oct 30 19:27:50.846227 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:50.846139    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bt8p\" (UniqueName: \"kubernetes.io/projected/3f52531a-7cb1-416c-86c2-61db7f8128d9-kube-api-access-9bt8p\") pod \"maas-api-revoke-sub-a476e910-t5dwj\" (UID: \"3f52531a-7cb1-416c-86c2-61db7f8128d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a476e910-t5dwj"
Oct 30 19:27:50.846933 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:50.846845    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f52531a-7cb1-416c-86c2-61db7f8128d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a476e910-t5dwj\" (UID: \"3f52531a-7cb1-416c-86c2-61db7f8128d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a476e910-t5dwj"
Oct 30 19:27:50.848694 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:50.848670    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f52531a-7cb1-416c-86c2-61db7f8128d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a476e910-t5dwj\" (UID: \"3f52531a-7cb1-416c-86c2-61db7f8128d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a476e910-t5dwj"
Oct 30 19:27:50.854256 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:50.854234    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bt8p\" (UniqueName: \"kubernetes.io/projected/3f52531a-7cb1-416c-86c2-61db7f8128d9-kube-api-access-9bt8p\") pod \"maas-api-revoke-sub-a476e910-t5dwj\" (UID: \"3f52531a-7cb1-416c-86c2-61db7f8128d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a476e910-t5dwj"
Oct 30 19:27:50.953146 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:50.953029    5043 generic.go:356] "Generic (PLEG): container finished" podID="93c44a79-bd6c-4c94-a74f-2cbf6d9240c0" containerID="9d33234436200af4ace587731a37c1290714fbc4785d4fdf4c1dda07ed4906c3" exitCode=0
Oct 30 19:27:50.953146 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:50.953124    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e16f494-qrpqh" event={"ID":"93c44a79-bd6c-4c94-a74f-2cbf6d9240c0","Type":"ContainerDied","Data":"9d33234436200af4ace587731a37c1290714fbc4785d4fdf4c1dda07ed4906c3"}
Oct 30 19:27:51.002722 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:51.002684    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a476e910-t5dwj"
Oct 30 19:27:51.143940 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:51.143912    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d1285b-drd4t"
Oct 30 19:27:51.251788 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:51.251749    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dec8f742-4e6e-4f2d-819f-a30304039c5b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dec8f742-4e6e-4f2d-819f-a30304039c5b-maas-api-xks-ca\") pod \"dec8f742-4e6e-4f2d-819f-a30304039c5b\" (UID: \"dec8f742-4e6e-4f2d-819f-a30304039c5b\") "
Oct 30 19:27:51.251788 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:51.251795    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dec8f742-4e6e-4f2d-819f-a30304039c5b-kube-api-access-dnvbp\" (UniqueName: \"kubernetes.io/projected/dec8f742-4e6e-4f2d-819f-a30304039c5b-kube-api-access-dnvbp\") pod \"dec8f742-4e6e-4f2d-819f-a30304039c5b\" (UID: \"dec8f742-4e6e-4f2d-819f-a30304039c5b\") "
Oct 30 19:27:51.252024 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:51.251925    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dec8f742-4e6e-4f2d-819f-a30304039c5b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dec8f742-4e6e-4f2d-819f-a30304039c5b-maas-api-openshift-ca\") pod \"dec8f742-4e6e-4f2d-819f-a30304039c5b\" (UID: \"dec8f742-4e6e-4f2d-819f-a30304039c5b\") "
Oct 30 19:27:51.252309 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:51.252283    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dec8f742-4e6e-4f2d-819f-a30304039c5b-maas-api-openshift-ca" pod "dec8f742-4e6e-4f2d-819f-a30304039c5b" (UID: "dec8f742-4e6e-4f2d-819f-a30304039c5b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:27:51.254159 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:51.254132    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dec8f742-4e6e-4f2d-819f-a30304039c5b-maas-api-xks-ca" pod "dec8f742-4e6e-4f2d-819f-a30304039c5b" (UID: "dec8f742-4e6e-4f2d-819f-a30304039c5b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:27:51.254245 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:51.254170    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dec8f742-4e6e-4f2d-819f-a30304039c5b-kube-api-access-dnvbp" pod "dec8f742-4e6e-4f2d-819f-a30304039c5b" (UID: "dec8f742-4e6e-4f2d-819f-a30304039c5b"). InnerVolumeSpecName "kube-api-access-dnvbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:27:51.298970 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:51.298938    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a476e910-t5dwj"]
Oct 30 19:27:51.299958 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:27:51.299924    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f52531a_7cb1_416c_86c2_61db7f8128d9.slice/crio-02e55c4c4b69990033b6d5b094a9bb89d38f466a91e294714014a4d8e35d0452 WatchSource:0}: Error finding container 02e55c4c4b69990033b6d5b094a9bb89d38f466a91e294714014a4d8e35d0452: Status 404 returned error can't find the container with id 02e55c4c4b69990033b6d5b094a9bb89d38f466a91e294714014a4d8e35d0452
Oct 30 19:27:51.353615 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:51.353587    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dec8f742-4e6e-4f2d-819f-a30304039c5b-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:51.353615 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:51.353614    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dec8f742-4e6e-4f2d-819f-a30304039c5b-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:51.353833 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:51.353625    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnvbp\" (UniqueName: \"kubernetes.io/projected/dec8f742-4e6e-4f2d-819f-a30304039c5b-kube-api-access-dnvbp\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:51.632394 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:51.632355    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-810f290e-pv6pg"]
Oct 30 19:27:51.632912 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:51.632898    5043 cpu_manager.go:414] "containerMap: removing container" podUID="dec8f742-4e6e-4f2d-819f-a30304039c5b" containerName="revoke-keys"
Oct 30 19:27:51.633015 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:51.632913    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dec8f742-4e6e-4f2d-819f-a30304039c5b" containerName="revoke-keys"
Oct 30 19:27:51.633015 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:51.633006    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="dec8f742-4e6e-4f2d-819f-a30304039c5b" containerName="revoke-keys"
Oct 30 19:27:51.637822 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:51.637800    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-810f290e-pv6pg"
Oct 30 19:27:51.645041 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:51.645014    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-810f290e-pv6pg"]
Oct 30 19:27:51.758968 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:51.758937    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13248701-d6e5-4fb3-bc91-28a028a9b4b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-810f290e-pv6pg\" (UID: \"13248701-d6e5-4fb3-bc91-28a028a9b4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-810f290e-pv6pg"
Oct 30 19:27:51.759149 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:51.758990    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2pbr\" (UniqueName: \"kubernetes.io/projected/13248701-d6e5-4fb3-bc91-28a028a9b4b3-kube-api-access-z2pbr\") pod \"maas-api-revoke-sub-810f290e-pv6pg\" (UID: \"13248701-d6e5-4fb3-bc91-28a028a9b4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-810f290e-pv6pg"
Oct 30 19:27:51.759380 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:51.759351    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13248701-d6e5-4fb3-bc91-28a028a9b4b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-810f290e-pv6pg\" (UID: \"13248701-d6e5-4fb3-bc91-28a028a9b4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-810f290e-pv6pg"
Oct 30 19:27:51.860317 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:51.860214    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13248701-d6e5-4fb3-bc91-28a028a9b4b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-810f290e-pv6pg\" (UID: \"13248701-d6e5-4fb3-bc91-28a028a9b4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-810f290e-pv6pg"
Oct 30 19:27:51.860489 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:51.860319    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13248701-d6e5-4fb3-bc91-28a028a9b4b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-810f290e-pv6pg\" (UID: \"13248701-d6e5-4fb3-bc91-28a028a9b4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-810f290e-pv6pg"
Oct 30 19:27:51.860489 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:51.860358    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2pbr\" (UniqueName: \"kubernetes.io/projected/13248701-d6e5-4fb3-bc91-28a028a9b4b3-kube-api-access-z2pbr\") pod \"maas-api-revoke-sub-810f290e-pv6pg\" (UID: \"13248701-d6e5-4fb3-bc91-28a028a9b4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-810f290e-pv6pg"
Oct 30 19:27:51.861116 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:51.861041    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13248701-d6e5-4fb3-bc91-28a028a9b4b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-810f290e-pv6pg\" (UID: \"13248701-d6e5-4fb3-bc91-28a028a9b4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-810f290e-pv6pg"
Oct 30 19:27:51.862868 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:51.862848    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13248701-d6e5-4fb3-bc91-28a028a9b4b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-810f290e-pv6pg\" (UID: \"13248701-d6e5-4fb3-bc91-28a028a9b4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-810f290e-pv6pg"
Oct 30 19:27:51.868291 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:51.868267    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2pbr\" (UniqueName: \"kubernetes.io/projected/13248701-d6e5-4fb3-bc91-28a028a9b4b3-kube-api-access-z2pbr\") pod \"maas-api-revoke-sub-810f290e-pv6pg\" (UID: \"13248701-d6e5-4fb3-bc91-28a028a9b4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-810f290e-pv6pg"
Oct 30 19:27:51.955048 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:51.955017    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-810f290e-pv6pg"
Oct 30 19:27:51.963309 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:51.962633    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a476e910-t5dwj" event={"ID":"3f52531a-7cb1-416c-86c2-61db7f8128d9","Type":"ContainerStarted","Data":"eb49954fe25115b44fae9855f478d44a1839f3f46f0db4a440de9202b7315f3f"}
Oct 30 19:27:51.963309 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:51.962692    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a476e910-t5dwj" event={"ID":"3f52531a-7cb1-416c-86c2-61db7f8128d9","Type":"ContainerStarted","Data":"02e55c4c4b69990033b6d5b094a9bb89d38f466a91e294714014a4d8e35d0452"}
Oct 30 19:27:51.964365 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:51.964340    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d1285b-drd4t"
Oct 30 19:27:51.964491 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:51.964339    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d1285b-drd4t" event={"ID":"dec8f742-4e6e-4f2d-819f-a30304039c5b","Type":"ContainerDied","Data":"aa8b5a609696d4919563577eb5976aa2998faae3e58b454c8edeba7dfcc0195e"}
Oct 30 19:27:51.964530 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:51.964513    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa8b5a609696d4919563577eb5976aa2998faae3e58b454c8edeba7dfcc0195e"
Oct 30 19:27:51.988771 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:51.986912    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a476e910-t5dwj" podStartSLOduration=1.986892627 podStartE2EDuration="1.986892627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:51.983190679 +0000 UTC m=+691.494250644" watchObservedRunningTime="2026-10-30 19:27:51.986892627 +0000 UTC m=+691.497952594"
Oct 30 19:27:52.165942 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:52.165912    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e16f494-qrpqh"
Oct 30 19:27:52.248933 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:52.248902    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-810f290e-pv6pg"]
Oct 30 19:27:52.249069 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:27:52.248946    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13248701_d6e5_4fb3_bc91_28a028a9b4b3.slice/crio-31a5bbb9183f659e6d1c39916d26f450b1e4e9babb935354241687548ab496b1 WatchSource:0}: Error finding container 31a5bbb9183f659e6d1c39916d26f450b1e4e9babb935354241687548ab496b1: Status 404 returned error can't find the container with id 31a5bbb9183f659e6d1c39916d26f450b1e4e9babb935354241687548ab496b1
Oct 30 19:27:52.264493 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:52.264469    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93c44a79-bd6c-4c94-a74f-2cbf6d9240c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93c44a79-bd6c-4c94-a74f-2cbf6d9240c0-maas-api-openshift-ca\") pod \"93c44a79-bd6c-4c94-a74f-2cbf6d9240c0\" (UID: \"93c44a79-bd6c-4c94-a74f-2cbf6d9240c0\") "
Oct 30 19:27:52.264874 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:52.264697    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93c44a79-bd6c-4c94-a74f-2cbf6d9240c0-kube-api-access-xdr6q\" (UniqueName: \"kubernetes.io/projected/93c44a79-bd6c-4c94-a74f-2cbf6d9240c0-kube-api-access-xdr6q\") pod \"93c44a79-bd6c-4c94-a74f-2cbf6d9240c0\" (UID: \"93c44a79-bd6c-4c94-a74f-2cbf6d9240c0\") "
Oct 30 19:27:52.264874 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:52.264759    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93c44a79-bd6c-4c94-a74f-2cbf6d9240c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93c44a79-bd6c-4c94-a74f-2cbf6d9240c0-maas-api-xks-ca\") pod \"93c44a79-bd6c-4c94-a74f-2cbf6d9240c0\" (UID: \"93c44a79-bd6c-4c94-a74f-2cbf6d9240c0\") "
Oct 30 19:27:52.266169 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:52.265332    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93c44a79-bd6c-4c94-a74f-2cbf6d9240c0-maas-api-openshift-ca" pod "93c44a79-bd6c-4c94-a74f-2cbf6d9240c0" (UID: "93c44a79-bd6c-4c94-a74f-2cbf6d9240c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:27:52.268245 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:52.268219    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93c44a79-bd6c-4c94-a74f-2cbf6d9240c0-maas-api-xks-ca" pod "93c44a79-bd6c-4c94-a74f-2cbf6d9240c0" (UID: "93c44a79-bd6c-4c94-a74f-2cbf6d9240c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:27:52.268752 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:52.268711    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93c44a79-bd6c-4c94-a74f-2cbf6d9240c0-kube-api-access-xdr6q" pod "93c44a79-bd6c-4c94-a74f-2cbf6d9240c0" (UID: "93c44a79-bd6c-4c94-a74f-2cbf6d9240c0"). InnerVolumeSpecName "kube-api-access-xdr6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:27:52.367387 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:52.367338    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdr6q\" (UniqueName: \"kubernetes.io/projected/93c44a79-bd6c-4c94-a74f-2cbf6d9240c0-kube-api-access-xdr6q\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:52.367541 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:52.367393    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93c44a79-bd6c-4c94-a74f-2cbf6d9240c0-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:52.367541 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:52.367410    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93c44a79-bd6c-4c94-a74f-2cbf6d9240c0-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:52.788623 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:52.788537    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-276b6327-nsqgx"]
Oct 30 19:27:52.789070 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:52.789055    5043 cpu_manager.go:414] "containerMap: removing container" podUID="93c44a79-bd6c-4c94-a74f-2cbf6d9240c0" containerName="revoke-keys"
Oct 30 19:27:52.789140 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:52.789071    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93c44a79-bd6c-4c94-a74f-2cbf6d9240c0" containerName="revoke-keys"
Oct 30 19:27:52.789199 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:52.789172    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="93c44a79-bd6c-4c94-a74f-2cbf6d9240c0" containerName="revoke-keys"
Oct 30 19:27:52.793468 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:52.793445    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-276b6327-nsqgx"
Oct 30 19:27:52.801730 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:52.801700    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-276b6327-nsqgx"]
Oct 30 19:27:52.873126 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:52.873090    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d84bdb9e-82ea-4a0f-a648-042208f16dac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-276b6327-nsqgx\" (UID: \"d84bdb9e-82ea-4a0f-a648-042208f16dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-276b6327-nsqgx"
Oct 30 19:27:52.873311 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:52.873198    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d84bdb9e-82ea-4a0f-a648-042208f16dac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-276b6327-nsqgx\" (UID: \"d84bdb9e-82ea-4a0f-a648-042208f16dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-276b6327-nsqgx"
Oct 30 19:27:52.873311 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:52.873247    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmrn7\" (UniqueName: \"kubernetes.io/projected/d84bdb9e-82ea-4a0f-a648-042208f16dac-kube-api-access-wmrn7\") pod \"maas-api-revoke-sub-276b6327-nsqgx\" (UID: \"d84bdb9e-82ea-4a0f-a648-042208f16dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-276b6327-nsqgx"
Oct 30 19:27:52.973249 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:52.973202    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-810f290e-pv6pg" event={"ID":"13248701-d6e5-4fb3-bc91-28a028a9b4b3","Type":"ContainerStarted","Data":"77480000ae1c5856dae25601d7feb64238e6ffb3619ddd3ff28beb70941d7b77"}
Oct 30 19:27:52.973725 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:52.973278    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-810f290e-pv6pg" event={"ID":"13248701-d6e5-4fb3-bc91-28a028a9b4b3","Type":"ContainerStarted","Data":"31a5bbb9183f659e6d1c39916d26f450b1e4e9babb935354241687548ab496b1"}
Oct 30 19:27:52.974072 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:52.974036    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d84bdb9e-82ea-4a0f-a648-042208f16dac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-276b6327-nsqgx\" (UID: \"d84bdb9e-82ea-4a0f-a648-042208f16dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-276b6327-nsqgx"
Oct 30 19:27:52.974188 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:52.974104    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d84bdb9e-82ea-4a0f-a648-042208f16dac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-276b6327-nsqgx\" (UID: \"d84bdb9e-82ea-4a0f-a648-042208f16dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-276b6327-nsqgx"
Oct 30 19:27:52.974188 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:52.974158    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmrn7\" (UniqueName: \"kubernetes.io/projected/d84bdb9e-82ea-4a0f-a648-042208f16dac-kube-api-access-wmrn7\") pod \"maas-api-revoke-sub-276b6327-nsqgx\" (UID: \"d84bdb9e-82ea-4a0f-a648-042208f16dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-276b6327-nsqgx"
Oct 30 19:27:52.974974 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:52.974950    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d84bdb9e-82ea-4a0f-a648-042208f16dac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-276b6327-nsqgx\" (UID: \"d84bdb9e-82ea-4a0f-a648-042208f16dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-276b6327-nsqgx"
Oct 30 19:27:52.975396 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:52.975371    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e16f494-qrpqh" event={"ID":"93c44a79-bd6c-4c94-a74f-2cbf6d9240c0","Type":"ContainerDied","Data":"a5ed59b7cef412fc21b7a1cfe51c05de12a800cbc560d2d3439d5e9d8af9a73d"}
Oct 30 19:27:52.975513 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:52.975419    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e16f494-qrpqh"
Oct 30 19:27:52.975513 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:52.975424    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5ed59b7cef412fc21b7a1cfe51c05de12a800cbc560d2d3439d5e9d8af9a73d"
Oct 30 19:27:52.977579 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:52.977552    5043 generic.go:356] "Generic (PLEG): container finished" podID="3f52531a-7cb1-416c-86c2-61db7f8128d9" containerID="eb49954fe25115b44fae9855f478d44a1839f3f46f0db4a440de9202b7315f3f" exitCode=0
Oct 30 19:27:52.977704 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:52.977641    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a476e910-t5dwj" event={"ID":"3f52531a-7cb1-416c-86c2-61db7f8128d9","Type":"ContainerDied","Data":"eb49954fe25115b44fae9855f478d44a1839f3f46f0db4a440de9202b7315f3f"}
Oct 30 19:27:52.978288 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:52.978269    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d84bdb9e-82ea-4a0f-a648-042208f16dac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-276b6327-nsqgx\" (UID: \"d84bdb9e-82ea-4a0f-a648-042208f16dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-276b6327-nsqgx"
Oct 30 19:27:52.983164 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:52.983144    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmrn7\" (UniqueName: \"kubernetes.io/projected/d84bdb9e-82ea-4a0f-a648-042208f16dac-kube-api-access-wmrn7\") pod \"maas-api-revoke-sub-276b6327-nsqgx\" (UID: \"d84bdb9e-82ea-4a0f-a648-042208f16dac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-276b6327-nsqgx"
Oct 30 19:27:52.994658 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:52.994617    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-810f290e-pv6pg" podStartSLOduration=1.99460492 podStartE2EDuration="1.99460492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:52.9912346 +0000 UTC m=+692.502294562" watchObservedRunningTime="2026-10-30 19:27:52.99460492 +0000 UTC m=+692.505664885"
Oct 30 19:27:53.114850 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:53.114771    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-276b6327-nsqgx"
Oct 30 19:27:53.411218 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:53.411180    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-276b6327-nsqgx"]
Oct 30 19:27:53.412331 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:27:53.412305    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd84bdb9e_82ea_4a0f_a648_042208f16dac.slice/crio-c31c451f0cc3472d94084c41070584c361bb7619f28c806e43a06713ca034985 WatchSource:0}: Error finding container c31c451f0cc3472d94084c41070584c361bb7619f28c806e43a06713ca034985: Status 404 returned error can't find the container with id c31c451f0cc3472d94084c41070584c361bb7619f28c806e43a06713ca034985
Oct 30 19:27:53.992085 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:53.992041    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-276b6327-nsqgx" event={"ID":"d84bdb9e-82ea-4a0f-a648-042208f16dac","Type":"ContainerStarted","Data":"576b8992df3a51a7e1e81961ed4457b7c19ce899d71e62254f117e2e667c6d61"}
Oct 30 19:27:53.992589 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:53.992108    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-276b6327-nsqgx" event={"ID":"d84bdb9e-82ea-4a0f-a648-042208f16dac","Type":"ContainerStarted","Data":"c31c451f0cc3472d94084c41070584c361bb7619f28c806e43a06713ca034985"}
Oct 30 19:27:53.993730 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:53.993703    5043 generic.go:356] "Generic (PLEG): container finished" podID="13248701-d6e5-4fb3-bc91-28a028a9b4b3" containerID="77480000ae1c5856dae25601d7feb64238e6ffb3619ddd3ff28beb70941d7b77" exitCode=0
Oct 30 19:27:53.993867 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:53.993787    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-810f290e-pv6pg" event={"ID":"13248701-d6e5-4fb3-bc91-28a028a9b4b3","Type":"ContainerDied","Data":"77480000ae1c5856dae25601d7feb64238e6ffb3619ddd3ff28beb70941d7b77"}
Oct 30 19:27:54.010904 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:54.010860    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-276b6327-nsqgx" podStartSLOduration=2.010846477 podStartE2EDuration="2.010846477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:54.008814273 +0000 UTC m=+693.519874239" watchObservedRunningTime="2026-10-30 19:27:54.010846477 +0000 UTC m=+693.521906443"
Oct 30 19:27:54.151309 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:54.151284    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a476e910-t5dwj"
Oct 30 19:27:54.195491 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:54.195460    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f52531a-7cb1-416c-86c2-61db7f8128d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f52531a-7cb1-416c-86c2-61db7f8128d9-maas-api-xks-ca\") pod \"3f52531a-7cb1-416c-86c2-61db7f8128d9\" (UID: \"3f52531a-7cb1-416c-86c2-61db7f8128d9\") "
Oct 30 19:27:54.195660 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:54.195551    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f52531a-7cb1-416c-86c2-61db7f8128d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f52531a-7cb1-416c-86c2-61db7f8128d9-maas-api-openshift-ca\") pod \"3f52531a-7cb1-416c-86c2-61db7f8128d9\" (UID: \"3f52531a-7cb1-416c-86c2-61db7f8128d9\") "
Oct 30 19:27:54.195660 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:54.195606    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f52531a-7cb1-416c-86c2-61db7f8128d9-kube-api-access-9bt8p\" (UniqueName: \"kubernetes.io/projected/3f52531a-7cb1-416c-86c2-61db7f8128d9-kube-api-access-9bt8p\") pod \"3f52531a-7cb1-416c-86c2-61db7f8128d9\" (UID: \"3f52531a-7cb1-416c-86c2-61db7f8128d9\") "
Oct 30 19:27:54.195995 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:54.195959    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f52531a-7cb1-416c-86c2-61db7f8128d9-maas-api-openshift-ca" pod "3f52531a-7cb1-416c-86c2-61db7f8128d9" (UID: "3f52531a-7cb1-416c-86c2-61db7f8128d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:27:54.196184 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:54.196165    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f52531a-7cb1-416c-86c2-61db7f8128d9-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:54.198162 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:54.198130    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f52531a-7cb1-416c-86c2-61db7f8128d9-maas-api-xks-ca" pod "3f52531a-7cb1-416c-86c2-61db7f8128d9" (UID: "3f52531a-7cb1-416c-86c2-61db7f8128d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:27:54.198275 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:54.198164    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f52531a-7cb1-416c-86c2-61db7f8128d9-kube-api-access-9bt8p" pod "3f52531a-7cb1-416c-86c2-61db7f8128d9" (UID: "3f52531a-7cb1-416c-86c2-61db7f8128d9"). InnerVolumeSpecName "kube-api-access-9bt8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:27:54.296981 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:54.296895    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f52531a-7cb1-416c-86c2-61db7f8128d9-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:54.296981 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:54.296927    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bt8p\" (UniqueName: \"kubernetes.io/projected/3f52531a-7cb1-416c-86c2-61db7f8128d9-kube-api-access-9bt8p\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:54.979279 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:27:54.979227    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:27:55.004083 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:55.004049    5043 generic.go:356] "Generic (PLEG): container finished" podID="d84bdb9e-82ea-4a0f-a648-042208f16dac" containerID="576b8992df3a51a7e1e81961ed4457b7c19ce899d71e62254f117e2e667c6d61" exitCode=0
Oct 30 19:27:55.004582 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:55.004127    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-276b6327-nsqgx" event={"ID":"d84bdb9e-82ea-4a0f-a648-042208f16dac","Type":"ContainerDied","Data":"576b8992df3a51a7e1e81961ed4457b7c19ce899d71e62254f117e2e667c6d61"}
Oct 30 19:27:55.005846 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:55.005823    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a476e910-t5dwj"
Oct 30 19:27:55.005971 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:55.005855    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a476e910-t5dwj" event={"ID":"3f52531a-7cb1-416c-86c2-61db7f8128d9","Type":"ContainerDied","Data":"02e55c4c4b69990033b6d5b094a9bb89d38f466a91e294714014a4d8e35d0452"}
Oct 30 19:27:55.005971 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:55.005891    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02e55c4c4b69990033b6d5b094a9bb89d38f466a91e294714014a4d8e35d0452"
Oct 30 19:27:55.170267 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:55.170244    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-810f290e-pv6pg"
Oct 30 19:27:55.210791 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:55.210755    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13248701-d6e5-4fb3-bc91-28a028a9b4b3-kube-api-access-z2pbr\" (UniqueName: \"kubernetes.io/projected/13248701-d6e5-4fb3-bc91-28a028a9b4b3-kube-api-access-z2pbr\") pod \"13248701-d6e5-4fb3-bc91-28a028a9b4b3\" (UID: \"13248701-d6e5-4fb3-bc91-28a028a9b4b3\") "
Oct 30 19:27:55.210941 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:55.210921    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13248701-d6e5-4fb3-bc91-28a028a9b4b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13248701-d6e5-4fb3-bc91-28a028a9b4b3-maas-api-xks-ca\") pod \"13248701-d6e5-4fb3-bc91-28a028a9b4b3\" (UID: \"13248701-d6e5-4fb3-bc91-28a028a9b4b3\") "
Oct 30 19:27:55.211123 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:55.211095    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13248701-d6e5-4fb3-bc91-28a028a9b4b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13248701-d6e5-4fb3-bc91-28a028a9b4b3-maas-api-openshift-ca\") pod \"13248701-d6e5-4fb3-bc91-28a028a9b4b3\" (UID: \"13248701-d6e5-4fb3-bc91-28a028a9b4b3\") "
Oct 30 19:27:55.211439 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:55.211405    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13248701-d6e5-4fb3-bc91-28a028a9b4b3-maas-api-openshift-ca" pod "13248701-d6e5-4fb3-bc91-28a028a9b4b3" (UID: "13248701-d6e5-4fb3-bc91-28a028a9b4b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:27:55.211574 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:55.211557    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13248701-d6e5-4fb3-bc91-28a028a9b4b3-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:55.213113 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:55.213084    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13248701-d6e5-4fb3-bc91-28a028a9b4b3-kube-api-access-z2pbr" pod "13248701-d6e5-4fb3-bc91-28a028a9b4b3" (UID: "13248701-d6e5-4fb3-bc91-28a028a9b4b3"). InnerVolumeSpecName "kube-api-access-z2pbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:27:55.213579 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:55.213560    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13248701-d6e5-4fb3-bc91-28a028a9b4b3-maas-api-xks-ca" pod "13248701-d6e5-4fb3-bc91-28a028a9b4b3" (UID: "13248701-d6e5-4fb3-bc91-28a028a9b4b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:27:55.312625 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:55.312534    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13248701-d6e5-4fb3-bc91-28a028a9b4b3-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:55.312625 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:55.312566    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2pbr\" (UniqueName: \"kubernetes.io/projected/13248701-d6e5-4fb3-bc91-28a028a9b4b3-kube-api-access-z2pbr\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:56.014937 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.014907    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-810f290e-pv6pg"
Oct 30 19:27:56.015371 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.014936    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-810f290e-pv6pg" event={"ID":"13248701-d6e5-4fb3-bc91-28a028a9b4b3","Type":"ContainerDied","Data":"31a5bbb9183f659e6d1c39916d26f450b1e4e9babb935354241687548ab496b1"}
Oct 30 19:27:56.015371 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.014996    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31a5bbb9183f659e6d1c39916d26f450b1e4e9babb935354241687548ab496b1"
Oct 30 19:27:56.188321 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.188293    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-276b6327-nsqgx"
Oct 30 19:27:56.221633 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.221598    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d84bdb9e-82ea-4a0f-a648-042208f16dac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d84bdb9e-82ea-4a0f-a648-042208f16dac-maas-api-openshift-ca\") pod \"d84bdb9e-82ea-4a0f-a648-042208f16dac\" (UID: \"d84bdb9e-82ea-4a0f-a648-042208f16dac\") "
Oct 30 19:27:56.221859 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.221645    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d84bdb9e-82ea-4a0f-a648-042208f16dac-kube-api-access-wmrn7\" (UniqueName: \"kubernetes.io/projected/d84bdb9e-82ea-4a0f-a648-042208f16dac-kube-api-access-wmrn7\") pod \"d84bdb9e-82ea-4a0f-a648-042208f16dac\" (UID: \"d84bdb9e-82ea-4a0f-a648-042208f16dac\") "
Oct 30 19:27:56.221859 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.221837    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d84bdb9e-82ea-4a0f-a648-042208f16dac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d84bdb9e-82ea-4a0f-a648-042208f16dac-maas-api-xks-ca\") pod \"d84bdb9e-82ea-4a0f-a648-042208f16dac\" (UID: \"d84bdb9e-82ea-4a0f-a648-042208f16dac\") "
Oct 30 19:27:56.221998 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.221944    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d84bdb9e-82ea-4a0f-a648-042208f16dac-maas-api-openshift-ca" pod "d84bdb9e-82ea-4a0f-a648-042208f16dac" (UID: "d84bdb9e-82ea-4a0f-a648-042208f16dac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:27:56.222709 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.222686    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d84bdb9e-82ea-4a0f-a648-042208f16dac-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:56.224010 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.223985    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d84bdb9e-82ea-4a0f-a648-042208f16dac-kube-api-access-wmrn7" pod "d84bdb9e-82ea-4a0f-a648-042208f16dac" (UID: "d84bdb9e-82ea-4a0f-a648-042208f16dac"). InnerVolumeSpecName "kube-api-access-wmrn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:27:56.224454 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.224436    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d84bdb9e-82ea-4a0f-a648-042208f16dac-maas-api-xks-ca" pod "d84bdb9e-82ea-4a0f-a648-042208f16dac" (UID: "d84bdb9e-82ea-4a0f-a648-042208f16dac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:27:56.324414 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.324330    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmrn7\" (UniqueName: \"kubernetes.io/projected/d84bdb9e-82ea-4a0f-a648-042208f16dac-kube-api-access-wmrn7\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:56.324414 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.324362    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d84bdb9e-82ea-4a0f-a648-042208f16dac-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:56.408827 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.408788    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"]
Oct 30 19:27:56.409341 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.409323    5043 cpu_manager.go:414] "containerMap: removing container" podUID="3f52531a-7cb1-416c-86c2-61db7f8128d9" containerName="revoke-keys"
Oct 30 19:27:56.409446 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.409343    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f52531a-7cb1-416c-86c2-61db7f8128d9" containerName="revoke-keys"
Oct 30 19:27:56.409446 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.409374    5043 cpu_manager.go:414] "containerMap: removing container" podUID="13248701-d6e5-4fb3-bc91-28a028a9b4b3" containerName="revoke-keys"
Oct 30 19:27:56.409446 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.409384    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13248701-d6e5-4fb3-bc91-28a028a9b4b3" containerName="revoke-keys"
Oct 30 19:27:56.409446 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.409415    5043 cpu_manager.go:414] "containerMap: removing container" podUID="d84bdb9e-82ea-4a0f-a648-042208f16dac" containerName="revoke-keys"
Oct 30 19:27:56.409446 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.409423    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d84bdb9e-82ea-4a0f-a648-042208f16dac" containerName="revoke-keys"
Oct 30 19:27:56.409682 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.409543    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="d84bdb9e-82ea-4a0f-a648-042208f16dac" containerName="revoke-keys"
Oct 30 19:27:56.409682 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.409562    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f52531a-7cb1-416c-86c2-61db7f8128d9" containerName="revoke-keys"
Oct 30 19:27:56.409682 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.409576    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="13248701-d6e5-4fb3-bc91-28a028a9b4b3" containerName="revoke-keys"
Oct 30 19:27:56.413199 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.413179    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:27:56.429721 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.429698    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ce69f3\"/\"distinct-w3-ce69f3-kserve-self-signed-certs\""
Oct 30 19:27:56.430027 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.429993    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"]
Oct 30 19:27:56.531115 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.528456    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c54b39a-1658-4bd9-aad0-5ec529126201-dshm\") pod \"distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp\" (UID: \"7c54b39a-1658-4bd9-aad0-5ec529126201\") " pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:27:56.531115 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.528518    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c54b39a-1658-4bd9-aad0-5ec529126201-tmp-dir\") pod \"distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp\" (UID: \"7c54b39a-1658-4bd9-aad0-5ec529126201\") " pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:27:56.531115 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.528609    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7nkm\" (UniqueName: \"kubernetes.io/projected/7c54b39a-1658-4bd9-aad0-5ec529126201-kube-api-access-r7nkm\") pod \"distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp\" (UID: \"7c54b39a-1658-4bd9-aad0-5ec529126201\") " pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:27:56.531115 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.528642    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c54b39a-1658-4bd9-aad0-5ec529126201-tls-certs\") pod \"distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp\" (UID: \"7c54b39a-1658-4bd9-aad0-5ec529126201\") " pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:27:56.531115 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.528668    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c54b39a-1658-4bd9-aad0-5ec529126201-model-cache\") pod \"distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp\" (UID: \"7c54b39a-1658-4bd9-aad0-5ec529126201\") " pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:27:56.531115 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.528769    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c54b39a-1658-4bd9-aad0-5ec529126201-home\") pod \"distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp\" (UID: \"7c54b39a-1658-4bd9-aad0-5ec529126201\") " pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:27:56.629700 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.629606    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c54b39a-1658-4bd9-aad0-5ec529126201-dshm\") pod \"distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp\" (UID: \"7c54b39a-1658-4bd9-aad0-5ec529126201\") " pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:27:56.629700 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.629648    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c54b39a-1658-4bd9-aad0-5ec529126201-tmp-dir\") pod \"distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp\" (UID: \"7c54b39a-1658-4bd9-aad0-5ec529126201\") " pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:27:56.629970 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.629703    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7nkm\" (UniqueName: \"kubernetes.io/projected/7c54b39a-1658-4bd9-aad0-5ec529126201-kube-api-access-r7nkm\") pod \"distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp\" (UID: \"7c54b39a-1658-4bd9-aad0-5ec529126201\") " pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:27:56.629970 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.629725    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c54b39a-1658-4bd9-aad0-5ec529126201-tls-certs\") pod \"distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp\" (UID: \"7c54b39a-1658-4bd9-aad0-5ec529126201\") " pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:27:56.629970 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.629775    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c54b39a-1658-4bd9-aad0-5ec529126201-model-cache\") pod \"distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp\" (UID: \"7c54b39a-1658-4bd9-aad0-5ec529126201\") " pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:27:56.629970 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.629831    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c54b39a-1658-4bd9-aad0-5ec529126201-home\") pod \"distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp\" (UID: \"7c54b39a-1658-4bd9-aad0-5ec529126201\") " pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:27:56.630158 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.630137    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c54b39a-1658-4bd9-aad0-5ec529126201-tmp-dir\") pod \"distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp\" (UID: \"7c54b39a-1658-4bd9-aad0-5ec529126201\") " pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:27:56.630225 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.630205    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c54b39a-1658-4bd9-aad0-5ec529126201-model-cache\") pod \"distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp\" (UID: \"7c54b39a-1658-4bd9-aad0-5ec529126201\") " pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:27:56.630273 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.630223    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c54b39a-1658-4bd9-aad0-5ec529126201-home\") pod \"distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp\" (UID: \"7c54b39a-1658-4bd9-aad0-5ec529126201\") " pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:27:56.632264 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.632238    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c54b39a-1658-4bd9-aad0-5ec529126201-dshm\") pod \"distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp\" (UID: \"7c54b39a-1658-4bd9-aad0-5ec529126201\") " pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:27:56.632412 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.632392    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c54b39a-1658-4bd9-aad0-5ec529126201-tls-certs\") pod \"distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp\" (UID: \"7c54b39a-1658-4bd9-aad0-5ec529126201\") " pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:27:56.639948 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.639923    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7nkm\" (UniqueName: \"kubernetes.io/projected/7c54b39a-1658-4bd9-aad0-5ec529126201-kube-api-access-r7nkm\") pod \"distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp\" (UID: \"7c54b39a-1658-4bd9-aad0-5ec529126201\") " pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:27:56.723528 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.723491    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c"]
Oct 30 19:27:56.725288 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.725264    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:27:56.728716 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.728698    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c"
Oct 30 19:27:56.760293 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.759715    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c"]
Oct 30 19:27:56.831322 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.831287    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-istio-data\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c\" (UID: \"33a5f4d9-fe3f-40ad-904d-db5ac720c81e\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c"
Oct 30 19:27:56.831484 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.831339    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-istio-token\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c\" (UID: \"33a5f4d9-fe3f-40ad-904d-db5ac720c81e\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c"
Oct 30 19:27:56.831569 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.831471    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-workload-certs\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c\" (UID: \"33a5f4d9-fe3f-40ad-904d-db5ac720c81e\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c"
Oct 30 19:27:56.831569 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.831512    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-istio-envoy\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c\" (UID: \"33a5f4d9-fe3f-40ad-904d-db5ac720c81e\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c"
Oct 30 19:27:56.831569 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.831550    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-istio-podinfo\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c\" (UID: \"33a5f4d9-fe3f-40ad-904d-db5ac720c81e\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c"
Oct 30 19:27:56.831718 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.831682    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nhhh\" (UniqueName: \"kubernetes.io/projected/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-kube-api-access-8nhhh\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c\" (UID: \"33a5f4d9-fe3f-40ad-904d-db5ac720c81e\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c"
Oct 30 19:27:56.831783 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.831768    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-workload-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c\" (UID: \"33a5f4d9-fe3f-40ad-904d-db5ac720c81e\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c"
Oct 30 19:27:56.831864 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.831849    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-istiod-ca-cert\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c\" (UID: \"33a5f4d9-fe3f-40ad-904d-db5ac720c81e\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c"
Oct 30 19:27:56.831904 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.831891    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-credential-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c\" (UID: \"33a5f4d9-fe3f-40ad-904d-db5ac720c81e\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c"
Oct 30 19:27:56.908845 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.908728    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc"]
Oct 30 19:27:56.912806 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.912788    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc"
Oct 30 19:27:56.917482 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.917464    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-ce69f3-openshift-default-dockercfg-ktjvt\""
Oct 30 19:27:56.925425 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.925396    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc"]
Oct 30 19:27:56.933035 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.933016    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-istio-podinfo\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c\" (UID: \"33a5f4d9-fe3f-40ad-904d-db5ac720c81e\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c"
Oct 30 19:27:56.933123 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.933067    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nhhh\" (UniqueName: \"kubernetes.io/projected/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-kube-api-access-8nhhh\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c\" (UID: \"33a5f4d9-fe3f-40ad-904d-db5ac720c81e\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c"
Oct 30 19:27:56.933123 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.933085    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-workload-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c\" (UID: \"33a5f4d9-fe3f-40ad-904d-db5ac720c81e\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c"
Oct 30 19:27:56.933193 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.933134    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-istiod-ca-cert\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c\" (UID: \"33a5f4d9-fe3f-40ad-904d-db5ac720c81e\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c"
Oct 30 19:27:56.933193 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.933159    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-credential-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c\" (UID: \"33a5f4d9-fe3f-40ad-904d-db5ac720c81e\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c"
Oct 30 19:27:56.933272 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.933211    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-istio-data\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c\" (UID: \"33a5f4d9-fe3f-40ad-904d-db5ac720c81e\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c"
Oct 30 19:27:56.933272 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.933240    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-istio-token\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c\" (UID: \"33a5f4d9-fe3f-40ad-904d-db5ac720c81e\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c"
Oct 30 19:27:56.933342 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.933281    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-workload-certs\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c\" (UID: \"33a5f4d9-fe3f-40ad-904d-db5ac720c81e\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c"
Oct 30 19:27:56.933342 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.933297    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-istio-envoy\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c\" (UID: \"33a5f4d9-fe3f-40ad-904d-db5ac720c81e\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c"
Oct 30 19:27:56.934041 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.933764    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-credential-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c\" (UID: \"33a5f4d9-fe3f-40ad-904d-db5ac720c81e\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c"
Oct 30 19:27:56.934041 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.933796    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-workload-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c\" (UID: \"33a5f4d9-fe3f-40ad-904d-db5ac720c81e\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c"
Oct 30 19:27:56.934192 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.934056    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-workload-certs\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c\" (UID: \"33a5f4d9-fe3f-40ad-904d-db5ac720c81e\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c"
Oct 30 19:27:56.934192 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.934090    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-istio-data\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c\" (UID: \"33a5f4d9-fe3f-40ad-904d-db5ac720c81e\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c"
Oct 30 19:27:56.934192 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.934167    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-istiod-ca-cert\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c\" (UID: \"33a5f4d9-fe3f-40ad-904d-db5ac720c81e\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c"
Oct 30 19:27:56.936196 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.936178    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-istio-envoy\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c\" (UID: \"33a5f4d9-fe3f-40ad-904d-db5ac720c81e\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c"
Oct 30 19:27:56.936522 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.936501    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-istio-podinfo\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c\" (UID: \"33a5f4d9-fe3f-40ad-904d-db5ac720c81e\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c"
Oct 30 19:27:56.942009 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.941983    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nhhh\" (UniqueName: \"kubernetes.io/projected/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-kube-api-access-8nhhh\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c\" (UID: \"33a5f4d9-fe3f-40ad-904d-db5ac720c81e\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c"
Oct 30 19:27:56.942956 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:56.942935    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-istio-token\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c\" (UID: \"33a5f4d9-fe3f-40ad-904d-db5ac720c81e\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c"
Oct 30 19:27:57.025261 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.025233    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"]
Oct 30 19:27:57.025792 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.025772    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-276b6327-nsqgx"
Oct 30 19:27:57.025918 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.025772    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-276b6327-nsqgx" event={"ID":"d84bdb9e-82ea-4a0f-a648-042208f16dac","Type":"ContainerDied","Data":"c31c451f0cc3472d94084c41070584c361bb7619f28c806e43a06713ca034985"}
Oct 30 19:27:57.025918 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.025896    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c31c451f0cc3472d94084c41070584c361bb7619f28c806e43a06713ca034985"
Oct 30 19:27:57.027020 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:27:57.026995    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c54b39a_1658_4bd9_aad0_5ec529126201.slice/crio-a19e2dd6e3f8ddc23c864d2efb4c822f20085623b9b803b8d4ef146c79191113 WatchSource:0}: Error finding container a19e2dd6e3f8ddc23c864d2efb4c822f20085623b9b803b8d4ef146c79191113: Status 404 returned error can't find the container with id a19e2dd6e3f8ddc23c864d2efb4c822f20085623b9b803b8d4ef146c79191113
Oct 30 19:27:57.034537 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.034516    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53142e30-10c0-413a-ae61-fb1b29f5fd10-istio-token\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc\" (UID: \"53142e30-10c0-413a-ae61-fb1b29f5fd10\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc"
Oct 30 19:27:57.034622 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.034548    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53142e30-10c0-413a-ae61-fb1b29f5fd10-workload-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc\" (UID: \"53142e30-10c0-413a-ae61-fb1b29f5fd10\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc"
Oct 30 19:27:57.034622 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.034575    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53142e30-10c0-413a-ae61-fb1b29f5fd10-credential-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc\" (UID: \"53142e30-10c0-413a-ae61-fb1b29f5fd10\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc"
Oct 30 19:27:57.034777 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.034725    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53142e30-10c0-413a-ae61-fb1b29f5fd10-istio-envoy\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc\" (UID: \"53142e30-10c0-413a-ae61-fb1b29f5fd10\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc"
Oct 30 19:27:57.034845 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.034816    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53142e30-10c0-413a-ae61-fb1b29f5fd10-istiod-ca-cert\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc\" (UID: \"53142e30-10c0-413a-ae61-fb1b29f5fd10\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc"
Oct 30 19:27:57.034898 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.034872    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53142e30-10c0-413a-ae61-fb1b29f5fd10-workload-certs\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc\" (UID: \"53142e30-10c0-413a-ae61-fb1b29f5fd10\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc"
Oct 30 19:27:57.034936 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.034918    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54fp6\" (UniqueName: \"kubernetes.io/projected/53142e30-10c0-413a-ae61-fb1b29f5fd10-kube-api-access-54fp6\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc\" (UID: \"53142e30-10c0-413a-ae61-fb1b29f5fd10\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc"
Oct 30 19:27:57.035054 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.035038    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53142e30-10c0-413a-ae61-fb1b29f5fd10-istio-data\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc\" (UID: \"53142e30-10c0-413a-ae61-fb1b29f5fd10\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc"
Oct 30 19:27:57.035095 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.035072    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53142e30-10c0-413a-ae61-fb1b29f5fd10-istio-podinfo\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc\" (UID: \"53142e30-10c0-413a-ae61-fb1b29f5fd10\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc"
Oct 30 19:27:57.067503 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.067465    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf"]
Oct 30 19:27:57.069020 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.068994    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c"
Oct 30 19:27:57.073979 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.073957    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf"
Oct 30 19:27:57.114657 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.114637    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf"]
Oct 30 19:27:57.139327 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.139299    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-istio-data\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf\" (UID: \"c5c180ba-4a6f-4fb5-9976-1bd4b3526081\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf"
Oct 30 19:27:57.139454 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.139340    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-credential-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf\" (UID: \"c5c180ba-4a6f-4fb5-9976-1bd4b3526081\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf"
Oct 30 19:27:57.139454 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.139362    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-istio-token\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf\" (UID: \"c5c180ba-4a6f-4fb5-9976-1bd4b3526081\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf"
Oct 30 19:27:57.139454 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.139399    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-workload-certs\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf\" (UID: \"c5c180ba-4a6f-4fb5-9976-1bd4b3526081\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf"
Oct 30 19:27:57.139454 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.139418    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53142e30-10c0-413a-ae61-fb1b29f5fd10-istio-token\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc\" (UID: \"53142e30-10c0-413a-ae61-fb1b29f5fd10\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc"
Oct 30 19:27:57.139454 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.139441    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53142e30-10c0-413a-ae61-fb1b29f5fd10-workload-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc\" (UID: \"53142e30-10c0-413a-ae61-fb1b29f5fd10\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc"
Oct 30 19:27:57.139613 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.139469    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53142e30-10c0-413a-ae61-fb1b29f5fd10-credential-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc\" (UID: \"53142e30-10c0-413a-ae61-fb1b29f5fd10\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc"
Oct 30 19:27:57.139613 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.139497    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-istio-podinfo\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf\" (UID: \"c5c180ba-4a6f-4fb5-9976-1bd4b3526081\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf"
Oct 30 19:27:57.139613 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.139520    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53142e30-10c0-413a-ae61-fb1b29f5fd10-istio-envoy\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc\" (UID: \"53142e30-10c0-413a-ae61-fb1b29f5fd10\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc"
Oct 30 19:27:57.139613 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.139534    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-istiod-ca-cert\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf\" (UID: \"c5c180ba-4a6f-4fb5-9976-1bd4b3526081\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf"
Oct 30 19:27:57.139613 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.139559    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53142e30-10c0-413a-ae61-fb1b29f5fd10-istiod-ca-cert\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc\" (UID: \"53142e30-10c0-413a-ae61-fb1b29f5fd10\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc"
Oct 30 19:27:57.139613 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.139574    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6sn9\" (UniqueName: \"kubernetes.io/projected/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-kube-api-access-l6sn9\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf\" (UID: \"c5c180ba-4a6f-4fb5-9976-1bd4b3526081\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf"
Oct 30 19:27:57.139613 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.139598    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-istio-envoy\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf\" (UID: \"c5c180ba-4a6f-4fb5-9976-1bd4b3526081\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf"
Oct 30 19:27:57.139867 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.139616    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-workload-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf\" (UID: \"c5c180ba-4a6f-4fb5-9976-1bd4b3526081\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf"
Oct 30 19:27:57.139867 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.139642    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53142e30-10c0-413a-ae61-fb1b29f5fd10-workload-certs\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc\" (UID: \"53142e30-10c0-413a-ae61-fb1b29f5fd10\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc"
Oct 30 19:27:57.139867 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.139657    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54fp6\" (UniqueName: \"kubernetes.io/projected/53142e30-10c0-413a-ae61-fb1b29f5fd10-kube-api-access-54fp6\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc\" (UID: \"53142e30-10c0-413a-ae61-fb1b29f5fd10\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc"
Oct 30 19:27:57.139867 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.139689    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53142e30-10c0-413a-ae61-fb1b29f5fd10-istio-data\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc\" (UID: \"53142e30-10c0-413a-ae61-fb1b29f5fd10\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc"
Oct 30 19:27:57.139867 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.139710    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53142e30-10c0-413a-ae61-fb1b29f5fd10-istio-podinfo\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc\" (UID: \"53142e30-10c0-413a-ae61-fb1b29f5fd10\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc"
Oct 30 19:27:57.140829 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.140803    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53142e30-10c0-413a-ae61-fb1b29f5fd10-istiod-ca-cert\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc\" (UID: \"53142e30-10c0-413a-ae61-fb1b29f5fd10\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc"
Oct 30 19:27:57.140980 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.140955    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53142e30-10c0-413a-ae61-fb1b29f5fd10-credential-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc\" (UID: \"53142e30-10c0-413a-ae61-fb1b29f5fd10\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc"
Oct 30 19:27:57.141101 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.141077    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53142e30-10c0-413a-ae61-fb1b29f5fd10-workload-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc\" (UID: \"53142e30-10c0-413a-ae61-fb1b29f5fd10\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc"
Oct 30 19:27:57.141377 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.141355    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53142e30-10c0-413a-ae61-fb1b29f5fd10-istio-data\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc\" (UID: \"53142e30-10c0-413a-ae61-fb1b29f5fd10\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc"
Oct 30 19:27:57.141476 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.141455    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53142e30-10c0-413a-ae61-fb1b29f5fd10-workload-certs\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc\" (UID: \"53142e30-10c0-413a-ae61-fb1b29f5fd10\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc"
Oct 30 19:27:57.142660 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.142638    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53142e30-10c0-413a-ae61-fb1b29f5fd10-istio-podinfo\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc\" (UID: \"53142e30-10c0-413a-ae61-fb1b29f5fd10\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc"
Oct 30 19:27:57.142836 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.142819    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53142e30-10c0-413a-ae61-fb1b29f5fd10-istio-envoy\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc\" (UID: \"53142e30-10c0-413a-ae61-fb1b29f5fd10\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc"
Oct 30 19:27:57.153303 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.153266    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53142e30-10c0-413a-ae61-fb1b29f5fd10-istio-token\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc\" (UID: \"53142e30-10c0-413a-ae61-fb1b29f5fd10\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc"
Oct 30 19:27:57.154339 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.154299    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54fp6\" (UniqueName: \"kubernetes.io/projected/53142e30-10c0-413a-ae61-fb1b29f5fd10-kube-api-access-54fp6\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc\" (UID: \"53142e30-10c0-413a-ae61-fb1b29f5fd10\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc"
Oct 30 19:27:57.228045 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.228016    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc"
Oct 30 19:27:57.242695 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.242352    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-istio-podinfo\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf\" (UID: \"c5c180ba-4a6f-4fb5-9976-1bd4b3526081\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf"
Oct 30 19:27:57.242695 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.242410    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-istiod-ca-cert\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf\" (UID: \"c5c180ba-4a6f-4fb5-9976-1bd4b3526081\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf"
Oct 30 19:27:57.242695 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.242452    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6sn9\" (UniqueName: \"kubernetes.io/projected/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-kube-api-access-l6sn9\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf\" (UID: \"c5c180ba-4a6f-4fb5-9976-1bd4b3526081\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf"
Oct 30 19:27:57.242695 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.242487    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-istio-envoy\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf\" (UID: \"c5c180ba-4a6f-4fb5-9976-1bd4b3526081\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf"
Oct 30 19:27:57.242695 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.242515    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-workload-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf\" (UID: \"c5c180ba-4a6f-4fb5-9976-1bd4b3526081\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf"
Oct 30 19:27:57.242695 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.242650    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-istio-data\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf\" (UID: \"c5c180ba-4a6f-4fb5-9976-1bd4b3526081\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf"
Oct 30 19:27:57.242695 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.242682    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-credential-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf\" (UID: \"c5c180ba-4a6f-4fb5-9976-1bd4b3526081\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf"
Oct 30 19:27:57.243174 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.242709    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-istio-token\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf\" (UID: \"c5c180ba-4a6f-4fb5-9976-1bd4b3526081\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf"
Oct 30 19:27:57.243174 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.242786    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-workload-certs\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf\" (UID: \"c5c180ba-4a6f-4fb5-9976-1bd4b3526081\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf"
Oct 30 19:27:57.243281 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.243188    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-workload-certs\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf\" (UID: \"c5c180ba-4a6f-4fb5-9976-1bd4b3526081\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf"
Oct 30 19:27:57.243934 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.243908    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-workload-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf\" (UID: \"c5c180ba-4a6f-4fb5-9976-1bd4b3526081\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf"
Oct 30 19:27:57.244523 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.244495    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-istiod-ca-cert\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf\" (UID: \"c5c180ba-4a6f-4fb5-9976-1bd4b3526081\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf"
Oct 30 19:27:57.244949 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.244924    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-credential-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf\" (UID: \"c5c180ba-4a6f-4fb5-9976-1bd4b3526081\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf"
Oct 30 19:27:57.245088 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.245068    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-istio-data\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf\" (UID: \"c5c180ba-4a6f-4fb5-9976-1bd4b3526081\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf"
Oct 30 19:27:57.246959 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.246934    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-istio-podinfo\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf\" (UID: \"c5c180ba-4a6f-4fb5-9976-1bd4b3526081\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf"
Oct 30 19:27:57.247162 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.247087    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-istio-envoy\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf\" (UID: \"c5c180ba-4a6f-4fb5-9976-1bd4b3526081\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf"
Oct 30 19:27:57.260073 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.260023    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6sn9\" (UniqueName: \"kubernetes.io/projected/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-kube-api-access-l6sn9\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf\" (UID: \"c5c180ba-4a6f-4fb5-9976-1bd4b3526081\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf"
Oct 30 19:27:57.264643 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.264596    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-istio-token\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf\" (UID: \"c5c180ba-4a6f-4fb5-9976-1bd4b3526081\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf"
Oct 30 19:27:57.392454 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.392423    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c"]
Oct 30 19:27:57.394013 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:27:57.393989    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33a5f4d9_fe3f_40ad_904d_db5ac720c81e.slice/crio-5b450fd6973a5e7c1332bd50bf684b29350cca0d1005082c2050f06333a53118 WatchSource:0}: Error finding container 5b450fd6973a5e7c1332bd50bf684b29350cca0d1005082c2050f06333a53118: Status 404 returned error can't find the container with id 5b450fd6973a5e7c1332bd50bf684b29350cca0d1005082c2050f06333a53118
Oct 30 19:27:57.397455 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.397412    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:27:57.397537 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.397510    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:27:57.397576 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.397555    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:27:57.426070 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.426046    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf"
Oct 30 19:27:57.548547 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.548504    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc"]
Oct 30 19:27:57.554226 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:27:57.554189    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53142e30_10c0_413a_ae61_fb1b29f5fd10.slice/crio-c09e32219273785bdbd5d58e66490410ee74b904defaba950163b3128a9c7e31 WatchSource:0}: Error finding container c09e32219273785bdbd5d58e66490410ee74b904defaba950163b3128a9c7e31: Status 404 returned error can't find the container with id c09e32219273785bdbd5d58e66490410ee74b904defaba950163b3128a9c7e31
Oct 30 19:27:57.556166 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.556134    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:27:57.556255 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.556202    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:27:57.556255 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.556232    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:27:57.732781 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.731916    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf"]
Oct 30 19:27:57.738838 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.738794    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:27:57.738969 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.738947    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:27:57.739074 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:57.739047    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:27:58.052823 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:58.049823    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68d0bbe2-qns88"]
Oct 30 19:27:58.072910 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:58.063973    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68d0bbe2-qns88"]
Oct 30 19:27:58.072910 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:58.064389    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d0bbe2-qns88"
Oct 30 19:27:58.108763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:58.076051    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:27:58.108763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:58.076205    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:27:58.108763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:58.076839    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:27:58.108763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:58.076931    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:27:58.173009 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:58.159862    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf" event={"ID":"c5c180ba-4a6f-4fb5-9976-1bd4b3526081","Type":"ContainerStarted","Data":"5479dc4f6f1bee401de9e545375dbdc8b111f8d453118eb9c4e84b4535fb8a02"}
Oct 30 19:27:58.173319 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:58.173288    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf" event={"ID":"c5c180ba-4a6f-4fb5-9976-1bd4b3526081","Type":"ContainerStarted","Data":"2ace47431e507256cda5e53f090b15021dd58d0f02f816db0c9e6572912123c5"}
Oct 30 19:27:58.173445 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:58.173431    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" event={"ID":"7c54b39a-1658-4bd9-aad0-5ec529126201","Type":"ContainerStarted","Data":"f52c942d70e971961ba7977e5fec2a34186a403576cc02998112ac2e906e9c87"}
Oct 30 19:27:58.173541 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:58.173529    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" event={"ID":"7c54b39a-1658-4bd9-aad0-5ec529126201","Type":"ContainerStarted","Data":"a19e2dd6e3f8ddc23c864d2efb4c822f20085623b9b803b8d4ef146c79191113"}
Oct 30 19:27:58.176391 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:58.176133    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4540da8d-6c20-4530-a264-b9b8ee80d890-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68d0bbe2-qns88\" (UID: \"4540da8d-6c20-4530-a264-b9b8ee80d890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d0bbe2-qns88"
Oct 30 19:27:58.176391 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:58.176320    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6654f\" (UniqueName: \"kubernetes.io/projected/4540da8d-6c20-4530-a264-b9b8ee80d890-kube-api-access-6654f\") pod \"maas-api-revoke-sub-68d0bbe2-qns88\" (UID: \"4540da8d-6c20-4530-a264-b9b8ee80d890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d0bbe2-qns88"
Oct 30 19:27:58.176391 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:58.176356    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4540da8d-6c20-4530-a264-b9b8ee80d890-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68d0bbe2-qns88\" (UID: \"4540da8d-6c20-4530-a264-b9b8ee80d890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d0bbe2-qns88"
Oct 30 19:27:58.178163 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:58.178084    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc" event={"ID":"53142e30-10c0-413a-ae61-fb1b29f5fd10","Type":"ContainerStarted","Data":"a3b8b859589480824dd49f3184380eb6d291f261a3f0e93a16262fc4419e998f"}
Oct 30 19:27:58.178163 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:58.178128    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc" event={"ID":"53142e30-10c0-413a-ae61-fb1b29f5fd10","Type":"ContainerStarted","Data":"c09e32219273785bdbd5d58e66490410ee74b904defaba950163b3128a9c7e31"}
Oct 30 19:27:58.188163 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:58.185818    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c" event={"ID":"33a5f4d9-fe3f-40ad-904d-db5ac720c81e","Type":"ContainerStarted","Data":"43dd5570ad53ac67d91af92c664436a5e37afb084da44e24a09abb05a57bfb81"}
Oct 30 19:27:58.188163 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:58.185882    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c" event={"ID":"33a5f4d9-fe3f-40ad-904d-db5ac720c81e","Type":"ContainerStarted","Data":"5b450fd6973a5e7c1332bd50bf684b29350cca0d1005082c2050f06333a53118"}
Oct 30 19:27:58.224087 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:58.224022    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf" podStartSLOduration=1.223999101 podStartE2EDuration="1.223999101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:58.195490273 +0000 UTC m=+697.706550243" watchObservedRunningTime="2026-10-30 19:27:58.223999101 +0000 UTC m=+697.735059066"
Oct 30 19:27:58.236041 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:58.236006    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc"
Oct 30 19:27:58.236226 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:58.236129    5043 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 19:27:58.236226 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:58.236171    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc" podUID="53142e30-10c0-413a-ae61-fb1b29f5fd10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 19:27:58.236430 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:58.236377    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podStartSLOduration=2.236358472 podStartE2EDuration="2.236358472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:58.233860481 +0000 UTC m=+697.744920447" watchObservedRunningTime="2026-10-30 19:27:58.236358472 +0000 UTC m=+697.747418438"
Oct 30 19:27:58.274764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:58.267424    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc" podStartSLOduration=2.267403666 podStartE2EDuration="2.267403666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:58.264211003 +0000 UTC m=+697.775270969" watchObservedRunningTime="2026-10-30 19:27:58.267403666 +0000 UTC m=+697.778463633"
Oct 30 19:27:58.289509 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:58.289471    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4540da8d-6c20-4530-a264-b9b8ee80d890-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68d0bbe2-qns88\" (UID: \"4540da8d-6c20-4530-a264-b9b8ee80d890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d0bbe2-qns88"
Oct 30 19:27:58.295189 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:58.295153    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6654f\" (UniqueName: \"kubernetes.io/projected/4540da8d-6c20-4530-a264-b9b8ee80d890-kube-api-access-6654f\") pod \"maas-api-revoke-sub-68d0bbe2-qns88\" (UID: \"4540da8d-6c20-4530-a264-b9b8ee80d890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d0bbe2-qns88"
Oct 30 19:27:58.295480 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:58.295457    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4540da8d-6c20-4530-a264-b9b8ee80d890-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68d0bbe2-qns88\" (UID: \"4540da8d-6c20-4530-a264-b9b8ee80d890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d0bbe2-qns88"
Oct 30 19:27:58.296465 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:58.296438    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4540da8d-6c20-4530-a264-b9b8ee80d890-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68d0bbe2-qns88\" (UID: \"4540da8d-6c20-4530-a264-b9b8ee80d890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d0bbe2-qns88"
Oct 30 19:27:58.317047 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:58.315084    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4540da8d-6c20-4530-a264-b9b8ee80d890-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68d0bbe2-qns88\" (UID: \"4540da8d-6c20-4530-a264-b9b8ee80d890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d0bbe2-qns88"
Oct 30 19:27:58.340968 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:58.340901    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6654f\" (UniqueName: \"kubernetes.io/projected/4540da8d-6c20-4530-a264-b9b8ee80d890-kube-api-access-6654f\") pod \"maas-api-revoke-sub-68d0bbe2-qns88\" (UID: \"4540da8d-6c20-4530-a264-b9b8ee80d890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d0bbe2-qns88"
Oct 30 19:27:58.428769 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:58.427082    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf"
Oct 30 19:27:58.429875 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:58.429298    5043 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 19:27:58.429875 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:58.429439    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf" podUID="c5c180ba-4a6f-4fb5-9976-1bd4b3526081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 19:27:58.484762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:58.476946    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d0bbe2-qns88"
Oct 30 19:27:58.733714 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:58.728874    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:27:59.073763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:59.073259    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c"
Oct 30 19:27:59.105764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:59.098094    5043 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 19:27:59.105764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:59.098149    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c" podUID="33a5f4d9-fe3f-40ad-904d-db5ac720c81e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 19:27:59.242767 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:59.239676    5043 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 19:27:59.242767 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:59.239763    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc" podUID="53142e30-10c0-413a-ae61-fb1b29f5fd10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 19:27:59.543402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:59.451129    5043 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 19:27:59.543402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:59.451182    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf" podUID="c5c180ba-4a6f-4fb5-9976-1bd4b3526081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 19:27:59.543402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:59.461266    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c" podStartSLOduration=3.461244477 podStartE2EDuration="3.461244477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:58.316016641 +0000 UTC m=+697.827076652" watchObservedRunningTime="2026-10-30 19:27:59.461244477 +0000 UTC m=+698.972304456"
Oct 30 19:27:59.543402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:59.462992    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68d0bbe2-qns88"]
Oct 30 19:27:59.890852 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:59.885749    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:27:59.967761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:27:59.958835    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:00.080347 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:00.080129    5043 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 19:28:00.081078 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:00.081036    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c" podUID="33a5f4d9-fe3f-40ad-904d-db5ac720c81e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 19:28:00.081373 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:00.081283    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:00.097454 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:00.097405    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:28:00.104287 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:00.098010    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:00.147148 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:00.147109    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:00.196884 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:00.196836    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:00.230203 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:00.230166    5043 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 19:28:00.230483 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:00.230449    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc" podUID="53142e30-10c0-413a-ae61-fb1b29f5fd10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 19:28:00.242763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:00.239263    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:00.429264 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:00.429170    5043 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 19:28:00.429542 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:00.429510    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf" podUID="c5c180ba-4a6f-4fb5-9976-1bd4b3526081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 19:28:00.556764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:00.554430    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d0bbe2-qns88" event={"ID":"4540da8d-6c20-4530-a264-b9b8ee80d890","Type":"ContainerStarted","Data":"e27482af84e53576b7cd04bbb54aa95f797bd6f6f9a98839b35bf514a6ad8b9f"}
Oct 30 19:28:00.581764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:00.577222    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d0bbe2-qns88" event={"ID":"4540da8d-6c20-4530-a264-b9b8ee80d890","Type":"ContainerStarted","Data":"d8640d96a021be4b1bcf5f887fceb28766336b426b547b0237db6735639a73a2"}
Oct 30 19:28:00.607049 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:00.606929    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d0bbe2-qns88" podStartSLOduration=3.606911089 podStartE2EDuration="3.606911089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:00.604864792 +0000 UTC m=+700.115924759" watchObservedRunningTime="2026-10-30 19:28:00.606911089 +0000 UTC m=+700.117971055"
Oct 30 19:28:00.746764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:00.743829    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:28:00.887763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:00.886702    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:00.887763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:00.887524    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:28:01.125159 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:01.090869    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:01.125159 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:01.091692    5043 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 19:28:01.125159 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:01.091721    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:28:01.125159 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:01.091759    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c" podUID="33a5f4d9-fe3f-40ad-904d-db5ac720c81e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 19:28:01.125159 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:01.107298    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:01.125159 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:01.108422    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:28:01.161816 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:01.147544    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:01.161816 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:01.148413    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:28:01.161816 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:01.149661    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:01.161816 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:01.150006    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:28:01.276819 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:01.242095    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:01.300770 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:01.249458    5043 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 19:28:01.340762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:01.301017    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc" podUID="53142e30-10c0-413a-ae61-fb1b29f5fd10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 19:28:01.483765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:01.315667    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:28:01.484534 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:01.479935    5043 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 19:28:01.484767 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:01.484719    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf" podUID="c5c180ba-4a6f-4fb5-9976-1bd4b3526081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 19:28:01.777780 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:01.756616    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:01.777780 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:01.776568    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:28:01.895764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:01.889356    5043 generic.go:356] "Generic (PLEG): container finished" podID="4540da8d-6c20-4530-a264-b9b8ee80d890" containerID="e27482af84e53576b7cd04bbb54aa95f797bd6f6f9a98839b35bf514a6ad8b9f" exitCode=0
Oct 30 19:28:01.906860 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:01.896000    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:01.924878 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:01.896043    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d0bbe2-qns88" event={"ID":"4540da8d-6c20-4530-a264-b9b8ee80d890","Type":"ContainerDied","Data":"e27482af84e53576b7cd04bbb54aa95f797bd6f6f9a98839b35bf514a6ad8b9f"}
Oct 30 19:28:02.098875 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:02.098782    5043 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 19:28:02.099121 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:02.099090    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c" podUID="33a5f4d9-fe3f-40ad-904d-db5ac720c81e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 19:28:02.125105 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:02.099795    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:28:02.125105 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:02.112001    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:28:02.155142 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:02.155097    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:28:02.156624 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:02.156571    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:28:02.244605 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:02.242876    5043 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 19:28:02.244605 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:02.242933    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc" podUID="53142e30-10c0-413a-ae61-fb1b29f5fd10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 19:28:02.304116 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:02.304071    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:28:02.433601 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:02.433505    5043 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 19:28:02.433905 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:02.433874    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf" podUID="c5c180ba-4a6f-4fb5-9976-1bd4b3526081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 19:28:02.460789 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:02.443760    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:28:02.847007 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:02.846949    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:02.926989 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:02.926944    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:28:03.190413 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:03.190312    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:03.225763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:03.191094    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:03.225763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:03.191611    5043 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 19:28:03.225763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:03.213005    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c" podUID="33a5f4d9-fe3f-40ad-904d-db5ac720c81e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 19:28:03.283396 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:03.283356    5043 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 19:28:03.283708 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:03.283660    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc" podUID="53142e30-10c0-413a-ae61-fb1b29f5fd10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 19:28:03.285856 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:03.285819    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:03.309955 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:03.308815    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:03.310578 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:03.310557    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:28:03.464819 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:03.450889    5043 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 19:28:03.464819 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:03.450972    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf" podUID="c5c180ba-4a6f-4fb5-9976-1bd4b3526081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 19:28:03.547569 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:03.547524    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:03.569156 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:03.569108    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:03.670107 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:03.670074    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:28:03.795952 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:03.795834    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:03.865014 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:03.864947    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:03.977460 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:03.976146    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:04.059029 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:04.001795    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:04.187030 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:04.174815    5043 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 19:28:04.187030 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:04.174873    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c" podUID="33a5f4d9-fe3f-40ad-904d-db5ac720c81e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 19:28:04.212047 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:04.212004    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:04.255167 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:04.236500    5043 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 19:28:04.255442 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:04.255410    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc" podUID="53142e30-10c0-413a-ae61-fb1b29f5fd10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 19:28:04.381074 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:04.380891    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:04.441680 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:04.441187    5043 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 19:28:04.441680 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:04.441250    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf" podUID="c5c180ba-4a6f-4fb5-9976-1bd4b3526081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 19:28:04.572073 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:04.572028    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:28:04.574101 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:04.574075    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:28:04.866096 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:04.866034    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:04.876794 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:04.872018    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d0bbe2-qns88"
Oct 30 19:28:04.986968 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:04.986909    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:05.094932 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:05.091886    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4540da8d-6c20-4530-a264-b9b8ee80d890-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4540da8d-6c20-4530-a264-b9b8ee80d890-maas-api-xks-ca\") pod \"4540da8d-6c20-4530-a264-b9b8ee80d890\" (UID: \"4540da8d-6c20-4530-a264-b9b8ee80d890\") "
Oct 30 19:28:05.094932 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:05.091978    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4540da8d-6c20-4530-a264-b9b8ee80d890-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4540da8d-6c20-4530-a264-b9b8ee80d890-maas-api-openshift-ca\") pod \"4540da8d-6c20-4530-a264-b9b8ee80d890\" (UID: \"4540da8d-6c20-4530-a264-b9b8ee80d890\") "
Oct 30 19:28:05.094932 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:05.092117    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4540da8d-6c20-4530-a264-b9b8ee80d890-kube-api-access-6654f\" (UniqueName: \"kubernetes.io/projected/4540da8d-6c20-4530-a264-b9b8ee80d890-kube-api-access-6654f\") pod \"4540da8d-6c20-4530-a264-b9b8ee80d890\" (UID: \"4540da8d-6c20-4530-a264-b9b8ee80d890\") "
Oct 30 19:28:05.094932 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:05.094369    5043 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 19:28:05.094932 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:05.094419    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c" podUID="33a5f4d9-fe3f-40ad-904d-db5ac720c81e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 19:28:05.122775 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:05.122237    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4540da8d-6c20-4530-a264-b9b8ee80d890-maas-api-openshift-ca" pod "4540da8d-6c20-4530-a264-b9b8ee80d890" (UID: "4540da8d-6c20-4530-a264-b9b8ee80d890"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:28:05.217763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:05.217197    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4540da8d-6c20-4530-a264-b9b8ee80d890-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:05.249820 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:05.249528    5043 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 19:28:05.249820 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:05.249602    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc" podUID="53142e30-10c0-413a-ae61-fb1b29f5fd10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 19:28:05.258044 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:05.257967    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:05.300026 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:05.276146    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4540da8d-6c20-4530-a264-b9b8ee80d890-maas-api-xks-ca" pod "4540da8d-6c20-4530-a264-b9b8ee80d890" (UID: "4540da8d-6c20-4530-a264-b9b8ee80d890"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:28:05.300528 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:05.300149    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:28:05.337038 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:05.337005    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4540da8d-6c20-4530-a264-b9b8ee80d890-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:05.345953 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:05.345895    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4540da8d-6c20-4530-a264-b9b8ee80d890-kube-api-access-6654f" pod "4540da8d-6c20-4530-a264-b9b8ee80d890" (UID: "4540da8d-6c20-4530-a264-b9b8ee80d890"). InnerVolumeSpecName "kube-api-access-6654f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:28:05.382949 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:05.381808    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:05.437780 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:05.436287    5043 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 19:28:05.437780 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:05.436350    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf" podUID="c5c180ba-4a6f-4fb5-9976-1bd4b3526081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 19:28:05.466002 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:05.464678    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6654f\" (UniqueName: \"kubernetes.io/projected/4540da8d-6c20-4530-a264-b9b8ee80d890-kube-api-access-6654f\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:05.523144 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:05.523052    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:28:05.576574 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:05.575816    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:28:05.763799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:05.753573    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d0bbe2-qns88" event={"ID":"4540da8d-6c20-4530-a264-b9b8ee80d890","Type":"ContainerDied","Data":"d8640d96a021be4b1bcf5f887fceb28766336b426b547b0237db6735639a73a2"}
Oct 30 19:28:05.763799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:05.753635    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8640d96a021be4b1bcf5f887fceb28766336b426b547b0237db6735639a73a2"
Oct 30 19:28:05.763799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:05.753704    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d0bbe2-qns88"
Oct 30 19:28:05.950956 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:05.950719    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:06.218880 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:06.213209    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:28:06.244010 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:06.243782    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:28:06.246283 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:06.246244    5043 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 19:28:06.246487 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:06.246466    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c" podUID="33a5f4d9-fe3f-40ad-904d-db5ac720c81e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 19:28:06.247366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:06.247346    5043 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 19:28:06.247531 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:06.247512    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc" podUID="53142e30-10c0-413a-ae61-fb1b29f5fd10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 19:28:06.322967 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:06.295660    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:28:06.324809 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:06.324783    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:28:06.437794 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:06.437751    5043 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 19:28:06.438150 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:06.438114    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf" podUID="c5c180ba-4a6f-4fb5-9976-1bd4b3526081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 19:28:06.737909 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:06.737863    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:28:06.768524 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:06.768494    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:28:06.798468 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:06.798419    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:28:07.222842 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:07.222801    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e33616c-znjxf"]
Oct 30 19:28:07.224203 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:07.224175    5043 cpu_manager.go:414] "containerMap: removing container" podUID="4540da8d-6c20-4530-a264-b9b8ee80d890" containerName="revoke-keys"
Oct 30 19:28:07.224354 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:07.224340    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4540da8d-6c20-4530-a264-b9b8ee80d890" containerName="revoke-keys"
Oct 30 19:28:07.224715 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:07.224699    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="4540da8d-6c20-4530-a264-b9b8ee80d890" containerName="revoke-keys"
Oct 30 19:28:07.233550 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:07.233522    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e33616c-znjxf"
Oct 30 19:28:07.242034 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:07.241970    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:28:07.246079 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:07.245903    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:28:07.246079 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:07.245998    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:28:07.246079 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:07.246025    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:28:07.247937 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:07.247107    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e33616c-znjxf"]
Oct 30 19:28:07.276179 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:07.275835    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mjjq\" (UniqueName: \"kubernetes.io/projected/cd13b35a-dad6-42a9-b694-af8475b38edc-kube-api-access-5mjjq\") pod \"maas-api-revoke-sub-1e33616c-znjxf\" (UID: \"cd13b35a-dad6-42a9-b694-af8475b38edc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e33616c-znjxf"
Oct 30 19:28:07.276179 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:07.275978    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd13b35a-dad6-42a9-b694-af8475b38edc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e33616c-znjxf\" (UID: \"cd13b35a-dad6-42a9-b694-af8475b38edc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e33616c-znjxf"
Oct 30 19:28:07.276179 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:07.276049    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd13b35a-dad6-42a9-b694-af8475b38edc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e33616c-znjxf\" (UID: \"cd13b35a-dad6-42a9-b694-af8475b38edc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e33616c-znjxf"
Oct 30 19:28:07.379305 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:07.379270    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd13b35a-dad6-42a9-b694-af8475b38edc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e33616c-znjxf\" (UID: \"cd13b35a-dad6-42a9-b694-af8475b38edc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e33616c-znjxf"
Oct 30 19:28:07.379656 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:07.379635    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd13b35a-dad6-42a9-b694-af8475b38edc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e33616c-znjxf\" (UID: \"cd13b35a-dad6-42a9-b694-af8475b38edc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e33616c-znjxf"
Oct 30 19:28:07.379898 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:07.379880    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mjjq\" (UniqueName: \"kubernetes.io/projected/cd13b35a-dad6-42a9-b694-af8475b38edc-kube-api-access-5mjjq\") pod \"maas-api-revoke-sub-1e33616c-znjxf\" (UID: \"cd13b35a-dad6-42a9-b694-af8475b38edc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e33616c-znjxf"
Oct 30 19:28:07.382007 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:07.381985    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd13b35a-dad6-42a9-b694-af8475b38edc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e33616c-znjxf\" (UID: \"cd13b35a-dad6-42a9-b694-af8475b38edc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e33616c-znjxf"
Oct 30 19:28:07.384988 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:07.384966    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd13b35a-dad6-42a9-b694-af8475b38edc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e33616c-znjxf\" (UID: \"cd13b35a-dad6-42a9-b694-af8475b38edc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e33616c-znjxf"
Oct 30 19:28:07.389435 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:07.389389    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mjjq\" (UniqueName: \"kubernetes.io/projected/cd13b35a-dad6-42a9-b694-af8475b38edc-kube-api-access-5mjjq\") pod \"maas-api-revoke-sub-1e33616c-znjxf\" (UID: \"cd13b35a-dad6-42a9-b694-af8475b38edc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e33616c-znjxf"
Oct 30 19:28:07.581013 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:07.580916    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e33616c-znjxf"
Oct 30 19:28:07.648853 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:07.647860    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf"
Oct 30 19:28:07.648853 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:07.648792    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf"
Oct 30 19:28:07.653623 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:07.653261    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf"
Oct 30 19:28:07.915347 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:07.915289    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e33616c-znjxf"]
Oct 30 19:28:08.113952 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:08.112123    5043 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:28:08.113952 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:08.112198    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c" podUID="33a5f4d9-fe3f-40ad-904d-db5ac720c81e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:08.274594 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:08.273904    5043 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:28:08.274594 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:08.273980    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc" podUID="53142e30-10c0-413a-ae61-fb1b29f5fd10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:08.479405 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:08.479368    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c"
Oct 30 19:28:08.481214 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:08.481103    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c"
Oct 30 19:28:08.482023 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:08.481974    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c"
Oct 30 19:28:08.494773 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:08.494646    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc"
Oct 30 19:28:08.495098 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:08.495078    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc"
Oct 30 19:28:08.497466 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:08.496636    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc"
Oct 30 19:28:08.806684 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:08.806648    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e33616c-znjxf" event={"ID":"cd13b35a-dad6-42a9-b694-af8475b38edc","Type":"ContainerStarted","Data":"2839512c12bdadf5b9a12229d9517f4f156dfadd6e531053fe955db4025c46e8"}
Oct 30 19:28:08.806886 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:08.806712    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e33616c-znjxf" event={"ID":"cd13b35a-dad6-42a9-b694-af8475b38edc","Type":"ContainerStarted","Data":"0c6929443c864f555735349e02097468e3a6c9547ca08a09aa1f0f633395f863"}
Oct 30 19:28:08.826545 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:08.826490    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e33616c-znjxf" podStartSLOduration=1.826474652 podStartE2EDuration="1.826474652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:08.822675902 +0000 UTC m=+708.333735880" watchObservedRunningTime="2026-10-30 19:28:08.826474652 +0000 UTC m=+708.337534618"
Oct 30 19:28:09.816048 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:09.816013    5043 generic.go:356] "Generic (PLEG): container finished" podID="cd13b35a-dad6-42a9-b694-af8475b38edc" containerID="2839512c12bdadf5b9a12229d9517f4f156dfadd6e531053fe955db4025c46e8" exitCode=0
Oct 30 19:28:09.816518 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:09.816100    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e33616c-znjxf" event={"ID":"cd13b35a-dad6-42a9-b694-af8475b38edc","Type":"ContainerDied","Data":"2839512c12bdadf5b9a12229d9517f4f156dfadd6e531053fe955db4025c46e8"}
Oct 30 19:28:11.006533 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.006505    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e33616c-znjxf"
Oct 30 19:28:11.069562 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.069524    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd13b35a-dad6-42a9-b694-af8475b38edc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd13b35a-dad6-42a9-b694-af8475b38edc-maas-api-xks-ca\") pod \"cd13b35a-dad6-42a9-b694-af8475b38edc\" (UID: \"cd13b35a-dad6-42a9-b694-af8475b38edc\") "
Oct 30 19:28:11.069854 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.069614    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd13b35a-dad6-42a9-b694-af8475b38edc-kube-api-access-5mjjq\" (UniqueName: \"kubernetes.io/projected/cd13b35a-dad6-42a9-b694-af8475b38edc-kube-api-access-5mjjq\") pod \"cd13b35a-dad6-42a9-b694-af8475b38edc\" (UID: \"cd13b35a-dad6-42a9-b694-af8475b38edc\") "
Oct 30 19:28:11.069854 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.069703    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd13b35a-dad6-42a9-b694-af8475b38edc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd13b35a-dad6-42a9-b694-af8475b38edc-maas-api-openshift-ca\") pod \"cd13b35a-dad6-42a9-b694-af8475b38edc\" (UID: \"cd13b35a-dad6-42a9-b694-af8475b38edc\") "
Oct 30 19:28:11.070211 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.070122    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd13b35a-dad6-42a9-b694-af8475b38edc-maas-api-openshift-ca" pod "cd13b35a-dad6-42a9-b694-af8475b38edc" (UID: "cd13b35a-dad6-42a9-b694-af8475b38edc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:28:11.070419 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.070397    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd13b35a-dad6-42a9-b694-af8475b38edc-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:11.079281 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.079211    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd13b35a-dad6-42a9-b694-af8475b38edc-maas-api-xks-ca" pod "cd13b35a-dad6-42a9-b694-af8475b38edc" (UID: "cd13b35a-dad6-42a9-b694-af8475b38edc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:28:11.079281 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.079233    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd13b35a-dad6-42a9-b694-af8475b38edc-kube-api-access-5mjjq" pod "cd13b35a-dad6-42a9-b694-af8475b38edc" (UID: "cd13b35a-dad6-42a9-b694-af8475b38edc"). InnerVolumeSpecName "kube-api-access-5mjjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:28:11.171926 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.171882    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd13b35a-dad6-42a9-b694-af8475b38edc-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:11.171926 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.171923    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mjjq\" (UniqueName: \"kubernetes.io/projected/cd13b35a-dad6-42a9-b694-af8475b38edc-kube-api-access-5mjjq\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:11.335396 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.335308    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf"]
Oct 30 19:28:11.335983 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.335961    5043 cpu_manager.go:414] "containerMap: removing container" podUID="cd13b35a-dad6-42a9-b694-af8475b38edc" containerName="revoke-keys"
Oct 30 19:28:11.335983 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.335983    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd13b35a-dad6-42a9-b694-af8475b38edc" containerName="revoke-keys"
Oct 30 19:28:11.336149 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.336115    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd13b35a-dad6-42a9-b694-af8475b38edc" containerName="revoke-keys"
Oct 30 19:28:11.339680 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.339661    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf"
Oct 30 19:28:11.363558 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.363522    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf"]
Oct 30 19:28:11.374100 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.374063    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fea9e2a-77c0-48c4-8517-3e104fc043b8-istio-podinfo\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf\" (UID: \"4fea9e2a-77c0-48c4-8517-3e104fc043b8\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf"
Oct 30 19:28:11.374380 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.374353    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fea9e2a-77c0-48c4-8517-3e104fc043b8-workload-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf\" (UID: \"4fea9e2a-77c0-48c4-8517-3e104fc043b8\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf"
Oct 30 19:28:11.374503 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.374466    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fea9e2a-77c0-48c4-8517-3e104fc043b8-workload-certs\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf\" (UID: \"4fea9e2a-77c0-48c4-8517-3e104fc043b8\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf"
Oct 30 19:28:11.374646 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.374619    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fea9e2a-77c0-48c4-8517-3e104fc043b8-istio-envoy\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf\" (UID: \"4fea9e2a-77c0-48c4-8517-3e104fc043b8\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf"
Oct 30 19:28:11.374715 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.374675    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzbfz\" (UniqueName: \"kubernetes.io/projected/4fea9e2a-77c0-48c4-8517-3e104fc043b8-kube-api-access-mzbfz\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf\" (UID: \"4fea9e2a-77c0-48c4-8517-3e104fc043b8\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf"
Oct 30 19:28:11.374803 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.374788    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fea9e2a-77c0-48c4-8517-3e104fc043b8-istio-token\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf\" (UID: \"4fea9e2a-77c0-48c4-8517-3e104fc043b8\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf"
Oct 30 19:28:11.374860 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.374823    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fea9e2a-77c0-48c4-8517-3e104fc043b8-istio-data\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf\" (UID: \"4fea9e2a-77c0-48c4-8517-3e104fc043b8\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf"
Oct 30 19:28:11.374924 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.374856    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fea9e2a-77c0-48c4-8517-3e104fc043b8-istiod-ca-cert\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf\" (UID: \"4fea9e2a-77c0-48c4-8517-3e104fc043b8\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf"
Oct 30 19:28:11.374924 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.374888    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fea9e2a-77c0-48c4-8517-3e104fc043b8-credential-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf\" (UID: \"4fea9e2a-77c0-48c4-8517-3e104fc043b8\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf"
Oct 30 19:28:11.476682 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.476647    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzbfz\" (UniqueName: \"kubernetes.io/projected/4fea9e2a-77c0-48c4-8517-3e104fc043b8-kube-api-access-mzbfz\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf\" (UID: \"4fea9e2a-77c0-48c4-8517-3e104fc043b8\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf"
Oct 30 19:28:11.477037 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.477002    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fea9e2a-77c0-48c4-8517-3e104fc043b8-istio-token\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf\" (UID: \"4fea9e2a-77c0-48c4-8517-3e104fc043b8\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf"
Oct 30 19:28:11.477142 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.477059    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fea9e2a-77c0-48c4-8517-3e104fc043b8-istio-data\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf\" (UID: \"4fea9e2a-77c0-48c4-8517-3e104fc043b8\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf"
Oct 30 19:28:11.477142 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.477087    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fea9e2a-77c0-48c4-8517-3e104fc043b8-istiod-ca-cert\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf\" (UID: \"4fea9e2a-77c0-48c4-8517-3e104fc043b8\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf"
Oct 30 19:28:11.477142 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.477134    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fea9e2a-77c0-48c4-8517-3e104fc043b8-credential-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf\" (UID: \"4fea9e2a-77c0-48c4-8517-3e104fc043b8\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf"
Oct 30 19:28:11.477279 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.477211    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fea9e2a-77c0-48c4-8517-3e104fc043b8-istio-podinfo\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf\" (UID: \"4fea9e2a-77c0-48c4-8517-3e104fc043b8\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf"
Oct 30 19:28:11.477279 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.477241    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fea9e2a-77c0-48c4-8517-3e104fc043b8-workload-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf\" (UID: \"4fea9e2a-77c0-48c4-8517-3e104fc043b8\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf"
Oct 30 19:28:11.477383 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.477346    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fea9e2a-77c0-48c4-8517-3e104fc043b8-workload-certs\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf\" (UID: \"4fea9e2a-77c0-48c4-8517-3e104fc043b8\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf"
Oct 30 19:28:11.477443 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.477426    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fea9e2a-77c0-48c4-8517-3e104fc043b8-istio-envoy\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf\" (UID: \"4fea9e2a-77c0-48c4-8517-3e104fc043b8\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf"
Oct 30 19:28:11.478588 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.478564    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fea9e2a-77c0-48c4-8517-3e104fc043b8-istiod-ca-cert\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf\" (UID: \"4fea9e2a-77c0-48c4-8517-3e104fc043b8\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf"
Oct 30 19:28:11.479468 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.479443    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fea9e2a-77c0-48c4-8517-3e104fc043b8-istio-data\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf\" (UID: \"4fea9e2a-77c0-48c4-8517-3e104fc043b8\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf"
Oct 30 19:28:11.480276 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.480253    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fea9e2a-77c0-48c4-8517-3e104fc043b8-workload-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf\" (UID: \"4fea9e2a-77c0-48c4-8517-3e104fc043b8\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf"
Oct 30 19:28:11.480671 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.480649    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fea9e2a-77c0-48c4-8517-3e104fc043b8-workload-certs\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf\" (UID: \"4fea9e2a-77c0-48c4-8517-3e104fc043b8\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf"
Oct 30 19:28:11.481164 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.481142    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fea9e2a-77c0-48c4-8517-3e104fc043b8-credential-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf\" (UID: \"4fea9e2a-77c0-48c4-8517-3e104fc043b8\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf"
Oct 30 19:28:11.487556 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.487525    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fea9e2a-77c0-48c4-8517-3e104fc043b8-istio-podinfo\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf\" (UID: \"4fea9e2a-77c0-48c4-8517-3e104fc043b8\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf"
Oct 30 19:28:11.487895 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.487871    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fea9e2a-77c0-48c4-8517-3e104fc043b8-istio-envoy\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf\" (UID: \"4fea9e2a-77c0-48c4-8517-3e104fc043b8\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf"
Oct 30 19:28:11.495551 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.495513    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fea9e2a-77c0-48c4-8517-3e104fc043b8-istio-token\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf\" (UID: \"4fea9e2a-77c0-48c4-8517-3e104fc043b8\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf"
Oct 30 19:28:11.495683 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.495655    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzbfz\" (UniqueName: \"kubernetes.io/projected/4fea9e2a-77c0-48c4-8517-3e104fc043b8-kube-api-access-mzbfz\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf\" (UID: \"4fea9e2a-77c0-48c4-8517-3e104fc043b8\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf"
Oct 30 19:28:11.655082 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.654990    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf"
Oct 30 19:28:11.831488 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.831456    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e33616c-znjxf"
Oct 30 19:28:11.831697 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.831486    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e33616c-znjxf" event={"ID":"cd13b35a-dad6-42a9-b694-af8475b38edc","Type":"ContainerDied","Data":"0c6929443c864f555735349e02097468e3a6c9547ca08a09aa1f0f633395f863"}
Oct 30 19:28:11.831697 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.831538    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c6929443c864f555735349e02097468e3a6c9547ca08a09aa1f0f633395f863"
Oct 30 19:28:11.948127 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.948093    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf"]
Oct 30 19:28:11.948832 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:28:11.948806    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fea9e2a_77c0_48c4_8517_3e104fc043b8.slice/crio-90b6d022465c3074cc0ac581da9058c50a0271f15227578f3b1e0ba5dc9f1a92 WatchSource:0}: Error finding container 90b6d022465c3074cc0ac581da9058c50a0271f15227578f3b1e0ba5dc9f1a92: Status 404 returned error can't find the container with id 90b6d022465c3074cc0ac581da9058c50a0271f15227578f3b1e0ba5dc9f1a92
Oct 30 19:28:11.951246 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.951215    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:28:11.951318 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.951279    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:28:11.951318 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:11.951308    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:28:12.841078 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:12.841041    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf" event={"ID":"4fea9e2a-77c0-48c4-8517-3e104fc043b8","Type":"ContainerStarted","Data":"a4d973530c2d7a358d00dfa3f48bfbea57b1ba2a3b9dd90f577c1b97cfccf368"}
Oct 30 19:28:12.841476 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:12.841104    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf" event={"ID":"4fea9e2a-77c0-48c4-8517-3e104fc043b8","Type":"ContainerStarted","Data":"90b6d022465c3074cc0ac581da9058c50a0271f15227578f3b1e0ba5dc9f1a92"}
Oct 30 19:28:12.896762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:12.896336    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf" podStartSLOduration=1.89631506 podStartE2EDuration="1.89631506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:12.89418732 +0000 UTC m=+712.405247286" watchObservedRunningTime="2026-10-30 19:28:12.89631506 +0000 UTC m=+712.407375030"
Oct 30 19:28:13.664760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:13.662075    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf"
Oct 30 19:28:13.664760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:13.662244    5043 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:28:13.664760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:13.662286    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf" podUID="4fea9e2a-77c0-48c4-8517-3e104fc043b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 19:28:14.315408 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:14.315372    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96"]
Oct 30 19:28:14.320684 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:14.320658    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96"
Oct 30 19:28:14.330628 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:14.330601    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-gt4zp\""
Oct 30 19:28:14.332294 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:14.332263    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96"]
Oct 30 19:28:14.417542 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:14.417497    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/377f3c78-650d-4680-aa0f-0373eba791eb-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qdh96\" (UID: \"377f3c78-650d-4680-aa0f-0373eba791eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96"
Oct 30 19:28:14.417773 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:14.417561    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/377f3c78-650d-4680-aa0f-0373eba791eb-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qdh96\" (UID: \"377f3c78-650d-4680-aa0f-0373eba791eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96"
Oct 30 19:28:14.417773 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:14.417593    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/377f3c78-650d-4680-aa0f-0373eba791eb-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qdh96\" (UID: \"377f3c78-650d-4680-aa0f-0373eba791eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96"
Oct 30 19:28:14.417773 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:14.417665    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/377f3c78-650d-4680-aa0f-0373eba791eb-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qdh96\" (UID: \"377f3c78-650d-4680-aa0f-0373eba791eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96"
Oct 30 19:28:14.417773 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:14.417715    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/377f3c78-650d-4680-aa0f-0373eba791eb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qdh96\" (UID: \"377f3c78-650d-4680-aa0f-0373eba791eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96"
Oct 30 19:28:14.417773 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:14.417760    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw87q\" (UniqueName: \"kubernetes.io/projected/377f3c78-650d-4680-aa0f-0373eba791eb-kube-api-access-cw87q\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qdh96\" (UID: \"377f3c78-650d-4680-aa0f-0373eba791eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96"
Oct 30 19:28:14.418038 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:14.417803    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/377f3c78-650d-4680-aa0f-0373eba791eb-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qdh96\" (UID: \"377f3c78-650d-4680-aa0f-0373eba791eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96"
Oct 30 19:28:14.418038 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:14.417864    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/377f3c78-650d-4680-aa0f-0373eba791eb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qdh96\" (UID: \"377f3c78-650d-4680-aa0f-0373eba791eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96"
Oct 30 19:28:14.418038 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:14.417908    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/377f3c78-650d-4680-aa0f-0373eba791eb-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qdh96\" (UID: \"377f3c78-650d-4680-aa0f-0373eba791eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96"
Oct 30 19:28:14.519078 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:14.519032    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/377f3c78-650d-4680-aa0f-0373eba791eb-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qdh96\" (UID: \"377f3c78-650d-4680-aa0f-0373eba791eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96"
Oct 30 19:28:14.519288 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:14.519089    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/377f3c78-650d-4680-aa0f-0373eba791eb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qdh96\" (UID: \"377f3c78-650d-4680-aa0f-0373eba791eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96"
Oct 30 19:28:14.519288 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:14.519110    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw87q\" (UniqueName: \"kubernetes.io/projected/377f3c78-650d-4680-aa0f-0373eba791eb-kube-api-access-cw87q\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qdh96\" (UID: \"377f3c78-650d-4680-aa0f-0373eba791eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96"
Oct 30 19:28:14.519288 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:14.519141    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/377f3c78-650d-4680-aa0f-0373eba791eb-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qdh96\" (UID: \"377f3c78-650d-4680-aa0f-0373eba791eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96"
Oct 30 19:28:14.519288 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:14.519178    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/377f3c78-650d-4680-aa0f-0373eba791eb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qdh96\" (UID: \"377f3c78-650d-4680-aa0f-0373eba791eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96"
Oct 30 19:28:14.519288 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:14.519214    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/377f3c78-650d-4680-aa0f-0373eba791eb-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qdh96\" (UID: \"377f3c78-650d-4680-aa0f-0373eba791eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96"
Oct 30 19:28:14.519581 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:14.519314    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/377f3c78-650d-4680-aa0f-0373eba791eb-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qdh96\" (UID: \"377f3c78-650d-4680-aa0f-0373eba791eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96"
Oct 30 19:28:14.519581 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:14.519355    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/377f3c78-650d-4680-aa0f-0373eba791eb-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qdh96\" (UID: \"377f3c78-650d-4680-aa0f-0373eba791eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96"
Oct 30 19:28:14.519581 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:14.519388    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/377f3c78-650d-4680-aa0f-0373eba791eb-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qdh96\" (UID: \"377f3c78-650d-4680-aa0f-0373eba791eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96"
Oct 30 19:28:14.519581 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:14.519513    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/377f3c78-650d-4680-aa0f-0373eba791eb-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qdh96\" (UID: \"377f3c78-650d-4680-aa0f-0373eba791eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96"
Oct 30 19:28:14.519851 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:14.519609    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/377f3c78-650d-4680-aa0f-0373eba791eb-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qdh96\" (UID: \"377f3c78-650d-4680-aa0f-0373eba791eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96"
Oct 30 19:28:14.519851 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:14.519776    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/377f3c78-650d-4680-aa0f-0373eba791eb-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qdh96\" (UID: \"377f3c78-650d-4680-aa0f-0373eba791eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96"
Oct 30 19:28:14.519851 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:14.519826    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/377f3c78-650d-4680-aa0f-0373eba791eb-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qdh96\" (UID: \"377f3c78-650d-4680-aa0f-0373eba791eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96"
Oct 30 19:28:14.520185 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:14.520162    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/377f3c78-650d-4680-aa0f-0373eba791eb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qdh96\" (UID: \"377f3c78-650d-4680-aa0f-0373eba791eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96"
Oct 30 19:28:14.521710 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:14.521691    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/377f3c78-650d-4680-aa0f-0373eba791eb-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qdh96\" (UID: \"377f3c78-650d-4680-aa0f-0373eba791eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96"
Oct 30 19:28:14.521855 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:14.521837    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/377f3c78-650d-4680-aa0f-0373eba791eb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qdh96\" (UID: \"377f3c78-650d-4680-aa0f-0373eba791eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96"
Oct 30 19:28:14.529903 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:14.529386    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw87q\" (UniqueName: \"kubernetes.io/projected/377f3c78-650d-4680-aa0f-0373eba791eb-kube-api-access-cw87q\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qdh96\" (UID: \"377f3c78-650d-4680-aa0f-0373eba791eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96"
Oct 30 19:28:14.536709 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:14.536680    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/377f3c78-650d-4680-aa0f-0373eba791eb-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qdh96\" (UID: \"377f3c78-650d-4680-aa0f-0373eba791eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96"
Oct 30 19:28:14.651963 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:14.651876    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96"
Oct 30 19:28:14.655592 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:14.655562    5043 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:28:14.655718 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:14.655602    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf" podUID="4fea9e2a-77c0-48c4-8517-3e104fc043b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 19:28:14.993625 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:14.993446    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96"]
Oct 30 19:28:14.995264 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:28:14.995231    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod377f3c78_650d_4680_aa0f_0373eba791eb.slice/crio-5df2da6cb055d8b4d01256b3f5cd7cb74229839f3fe6921534417630dd8983f3 WatchSource:0}: Error finding container 5df2da6cb055d8b4d01256b3f5cd7cb74229839f3fe6921534417630dd8983f3: Status 404 returned error can't find the container with id 5df2da6cb055d8b4d01256b3f5cd7cb74229839f3fe6921534417630dd8983f3
Oct 30 19:28:14.998570 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:14.998512    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:28:14.998661 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:14.998632    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:28:14.998756 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:14.998692    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:28:15.655643 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:15.655609    5043 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:28:15.656153 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:15.655665    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf" podUID="4fea9e2a-77c0-48c4-8517-3e104fc043b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 19:28:15.890405 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:15.889188    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96" event={"ID":"377f3c78-650d-4680-aa0f-0373eba791eb","Type":"ContainerStarted","Data":"7f2abe2aab6157719288b52cb9c9cf58ac9724e3b060f8c88bf92e623d1784fa"}
Oct 30 19:28:15.890405 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:15.889294    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96" event={"ID":"377f3c78-650d-4680-aa0f-0373eba791eb","Type":"ContainerStarted","Data":"5df2da6cb055d8b4d01256b3f5cd7cb74229839f3fe6921534417630dd8983f3"}
Oct 30 19:28:15.934833 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:15.934703    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96" podStartSLOduration=1.934688822 podStartE2EDuration="1.934688822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:15.93053884 +0000 UTC m=+715.441598804" watchObservedRunningTime="2026-10-30 19:28:15.934688822 +0000 UTC m=+715.445748801"
Oct 30 19:28:16.452550 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:16.452519    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8034062d-vrdkc"]
Oct 30 19:28:16.486758 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:16.484029    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8034062d-vrdkc"
Oct 30 19:28:16.504775 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:16.503056    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8034062d-vrdkc"]
Oct 30 19:28:16.532125 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:16.523077    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:28:16.532125 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:16.523206    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:28:16.532125 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:16.523239    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:28:16.532125 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:16.523461    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:28:16.551817 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:16.551774    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3ad1c21-2943-40d3-bfd8-b206e6ff488f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8034062d-vrdkc\" (UID: \"e3ad1c21-2943-40d3-bfd8-b206e6ff488f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8034062d-vrdkc"
Oct 30 19:28:16.552003 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:16.551864    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb4p8\" (UniqueName: \"kubernetes.io/projected/e3ad1c21-2943-40d3-bfd8-b206e6ff488f-kube-api-access-hb4p8\") pod \"maas-api-revoke-sub-8034062d-vrdkc\" (UID: \"e3ad1c21-2943-40d3-bfd8-b206e6ff488f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8034062d-vrdkc"
Oct 30 19:28:16.552067 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:16.552041    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3ad1c21-2943-40d3-bfd8-b206e6ff488f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8034062d-vrdkc\" (UID: \"e3ad1c21-2943-40d3-bfd8-b206e6ff488f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8034062d-vrdkc"
Oct 30 19:28:16.652441 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:16.652406    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96"
Oct 30 19:28:16.653004 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:16.652983    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3ad1c21-2943-40d3-bfd8-b206e6ff488f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8034062d-vrdkc\" (UID: \"e3ad1c21-2943-40d3-bfd8-b206e6ff488f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8034062d-vrdkc"
Oct 30 19:28:16.653093 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:16.653057    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb4p8\" (UniqueName: \"kubernetes.io/projected/e3ad1c21-2943-40d3-bfd8-b206e6ff488f-kube-api-access-hb4p8\") pod \"maas-api-revoke-sub-8034062d-vrdkc\" (UID: \"e3ad1c21-2943-40d3-bfd8-b206e6ff488f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8034062d-vrdkc"
Oct 30 19:28:16.653631 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:16.653616    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3ad1c21-2943-40d3-bfd8-b206e6ff488f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8034062d-vrdkc\" (UID: \"e3ad1c21-2943-40d3-bfd8-b206e6ff488f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8034062d-vrdkc"
Oct 30 19:28:16.654100 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:16.654077    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qdh96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:28:16.654163 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:16.654112    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96" podUID="377f3c78-650d-4680-aa0f-0373eba791eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 19:28:16.654351 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:16.653726    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3ad1c21-2943-40d3-bfd8-b206e6ff488f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8034062d-vrdkc\" (UID: \"e3ad1c21-2943-40d3-bfd8-b206e6ff488f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8034062d-vrdkc"
Oct 30 19:28:16.655504 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:16.655488    5043 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:28:16.655559 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:16.655525    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf" podUID="4fea9e2a-77c0-48c4-8517-3e104fc043b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 19:28:16.703816 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:16.687211    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3ad1c21-2943-40d3-bfd8-b206e6ff488f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8034062d-vrdkc\" (UID: \"e3ad1c21-2943-40d3-bfd8-b206e6ff488f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8034062d-vrdkc"
Oct 30 19:28:16.717766 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:16.716818    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb4p8\" (UniqueName: \"kubernetes.io/projected/e3ad1c21-2943-40d3-bfd8-b206e6ff488f-kube-api-access-hb4p8\") pod \"maas-api-revoke-sub-8034062d-vrdkc\" (UID: \"e3ad1c21-2943-40d3-bfd8-b206e6ff488f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8034062d-vrdkc"
Oct 30 19:28:16.842320 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:16.842257    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8034062d-vrdkc"
Oct 30 19:28:17.663829 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:17.659409    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qdh96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:28:17.663829 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:17.659472    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96" podUID="377f3c78-650d-4680-aa0f-0373eba791eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 19:28:17.752917 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:17.744429    5043 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:28:17.752917 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:17.744489    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf" podUID="4fea9e2a-77c0-48c4-8517-3e104fc043b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 19:28:18.139761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:18.129986    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8034062d-vrdkc"]
Oct 30 19:28:18.172313 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:28:18.172276    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3ad1c21_2943_40d3_bfd8_b206e6ff488f.slice/crio-bd9c08e690d7366c6d5b88f942fa2442d276e01327040395a924f2157f8fe126 WatchSource:0}: Error finding container bd9c08e690d7366c6d5b88f942fa2442d276e01327040395a924f2157f8fe126: Status 404 returned error can't find the container with id bd9c08e690d7366c6d5b88f942fa2442d276e01327040395a924f2157f8fe126
Oct 30 19:28:18.225868 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:18.225826    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:28:18.232193 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:18.226069    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:18.325967 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:18.325855    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:28:18.628779 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:18.628639    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:18.679192 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:18.671383    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qdh96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:28:18.679192 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:18.671449    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96" podUID="377f3c78-650d-4680-aa0f-0373eba791eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 19:28:18.679192 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:18.672630    5043 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:28:18.679192 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:18.672671    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf" podUID="4fea9e2a-77c0-48c4-8517-3e104fc043b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 19:28:18.795954 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:18.794874    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:18.869956 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:18.869848    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:28:18.937882 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:18.937831    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:28:19.099797 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:19.094240    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded"
Oct 30 19:28:19.111699 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:19.111663    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8034062d-vrdkc" event={"ID":"e3ad1c21-2943-40d3-bfd8-b206e6ff488f","Type":"ContainerStarted","Data":"bd9c08e690d7366c6d5b88f942fa2442d276e01327040395a924f2157f8fe126"}
Oct 30 19:28:19.251764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:19.233821    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:28:19.251764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:19.234754    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:28:19.251764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:19.245852    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:19.251764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:19.251259    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:28:19.332763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:19.321502    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:19.452155 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:19.409176    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:28:19.452155 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:19.409628    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:28:19.491834 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:19.473509    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:28:19.654754 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:19.654011    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qdh96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:28:19.654754 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:19.654068    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96" podUID="377f3c78-650d-4680-aa0f-0373eba791eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 19:28:19.655257 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:19.655233    5043 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:28:19.655399 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:19.655380    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf" podUID="4fea9e2a-77c0-48c4-8517-3e104fc043b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 19:28:19.728343 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:19.728291    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:28:19.853823 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:19.853710    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:28:20.125969 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:20.125012    5043 generic.go:356] "Generic (PLEG): container finished" podID="e3ad1c21-2943-40d3-bfd8-b206e6ff488f" containerID="c6ed548dd294d66f989d76d4d629835e6f7829db8444c3d77b7c867b2f139eb9" exitCode=0
Oct 30 19:28:20.125969 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:20.125104    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8034062d-vrdkc" event={"ID":"e3ad1c21-2943-40d3-bfd8-b206e6ff488f","Type":"ContainerDied","Data":"c6ed548dd294d66f989d76d4d629835e6f7829db8444c3d77b7c867b2f139eb9"}
Oct 30 19:28:20.652605 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:20.652573    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qdh96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:28:20.652827 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:20.652629    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96" podUID="377f3c78-650d-4680-aa0f-0373eba791eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 19:28:21.515227 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:21.514826    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8034062d-vrdkc"
Oct 30 19:28:21.638224 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:21.638188    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3ad1c21-2943-40d3-bfd8-b206e6ff488f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3ad1c21-2943-40d3-bfd8-b206e6ff488f-maas-api-xks-ca\") pod \"e3ad1c21-2943-40d3-bfd8-b206e6ff488f\" (UID: \"e3ad1c21-2943-40d3-bfd8-b206e6ff488f\") "
Oct 30 19:28:21.638412 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:21.638332    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3ad1c21-2943-40d3-bfd8-b206e6ff488f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3ad1c21-2943-40d3-bfd8-b206e6ff488f-maas-api-openshift-ca\") pod \"e3ad1c21-2943-40d3-bfd8-b206e6ff488f\" (UID: \"e3ad1c21-2943-40d3-bfd8-b206e6ff488f\") "
Oct 30 19:28:21.638412 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:21.638367    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3ad1c21-2943-40d3-bfd8-b206e6ff488f-kube-api-access-hb4p8\" (UniqueName: \"kubernetes.io/projected/e3ad1c21-2943-40d3-bfd8-b206e6ff488f-kube-api-access-hb4p8\") pod \"e3ad1c21-2943-40d3-bfd8-b206e6ff488f\" (UID: \"e3ad1c21-2943-40d3-bfd8-b206e6ff488f\") "
Oct 30 19:28:21.639625 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:21.639597    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3ad1c21-2943-40d3-bfd8-b206e6ff488f-maas-api-openshift-ca" pod "e3ad1c21-2943-40d3-bfd8-b206e6ff488f" (UID: "e3ad1c21-2943-40d3-bfd8-b206e6ff488f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:28:21.642345 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:21.641855    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3ad1c21-2943-40d3-bfd8-b206e6ff488f-kube-api-access-hb4p8" pod "e3ad1c21-2943-40d3-bfd8-b206e6ff488f" (UID: "e3ad1c21-2943-40d3-bfd8-b206e6ff488f"). InnerVolumeSpecName "kube-api-access-hb4p8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:28:21.642977 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:21.642944    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3ad1c21-2943-40d3-bfd8-b206e6ff488f-maas-api-xks-ca" pod "e3ad1c21-2943-40d3-bfd8-b206e6ff488f" (UID: "e3ad1c21-2943-40d3-bfd8-b206e6ff488f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:28:21.652368 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:21.652331    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qdh96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:28:21.652496 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:21.652401    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96" podUID="377f3c78-650d-4680-aa0f-0373eba791eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 19:28:21.692819 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:21.689762    5043 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:28:21.692819 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:21.689854    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf" podUID="4fea9e2a-77c0-48c4-8517-3e104fc043b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:21.740563 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:21.740529    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3ad1c21-2943-40d3-bfd8-b206e6ff488f-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:21.740822 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:21.740798    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3ad1c21-2943-40d3-bfd8-b206e6ff488f-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:21.740943 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:21.740827    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb4p8\" (UniqueName: \"kubernetes.io/projected/e3ad1c21-2943-40d3-bfd8-b206e6ff488f-kube-api-access-hb4p8\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:21.780764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:21.779311    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf"
Oct 30 19:28:21.780941 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:21.780902    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf"
Oct 30 19:28:21.788763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:21.783971    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf"
Oct 30 19:28:22.144141 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:22.144114    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8034062d-vrdkc"
Oct 30 19:28:22.144141 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:22.144127    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8034062d-vrdkc" event={"ID":"e3ad1c21-2943-40d3-bfd8-b206e6ff488f","Type":"ContainerDied","Data":"bd9c08e690d7366c6d5b88f942fa2442d276e01327040395a924f2157f8fe126"}
Oct 30 19:28:22.144410 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:22.144175    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd9c08e690d7366c6d5b88f942fa2442d276e01327040395a924f2157f8fe126"
Oct 30 19:28:22.653101 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:22.653066    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qdh96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:28:22.653571 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:22.653119    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96" podUID="377f3c78-650d-4680-aa0f-0373eba791eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 19:28:23.653285 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:23.653238    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qdh96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:28:23.653832 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:23.653312    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96" podUID="377f3c78-650d-4680-aa0f-0373eba791eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 19:28:24.986187 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:28:24.986108    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:28:25.183334 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:25.183298    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96"
Oct 30 19:28:25.183830 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:25.183799    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96"
Oct 30 19:28:25.185040 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:25.185017    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96"
Oct 30 19:28:26.592621 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.592586    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt"]
Oct 30 19:28:26.593152 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.593144    5043 cpu_manager.go:414] "containerMap: removing container" podUID="e3ad1c21-2943-40d3-bfd8-b206e6ff488f" containerName="revoke-keys"
Oct 30 19:28:26.593220 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.593156    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3ad1c21-2943-40d3-bfd8-b206e6ff488f" containerName="revoke-keys"
Oct 30 19:28:26.593281 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.593271    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3ad1c21-2943-40d3-bfd8-b206e6ff488f" containerName="revoke-keys"
Oct 30 19:28:26.598278 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.598239    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt"
Oct 30 19:28:26.609904 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.609878    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt"]
Oct 30 19:28:26.674232 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.674200    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-768af474-j5578"]
Oct 30 19:28:26.678404 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.678379    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768af474-j5578"
Oct 30 19:28:26.682934 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.682889    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:28:26.683117 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.682988    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:28:26.683117 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.683061    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:28:26.683117 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.683093    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:28:26.688771 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.688713    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-768af474-j5578"]
Oct 30 19:28:26.715178 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.715147    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/359c47fc-f8af-43f8-865b-1f187fb2672c-istiod-ca-cert\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt\" (UID: \"359c47fc-f8af-43f8-865b-1f187fb2672c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt"
Oct 30 19:28:26.715383 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.715185    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/359c47fc-f8af-43f8-865b-1f187fb2672c-istio-data\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt\" (UID: \"359c47fc-f8af-43f8-865b-1f187fb2672c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt"
Oct 30 19:28:26.715383 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.715221    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/359c47fc-f8af-43f8-865b-1f187fb2672c-istio-token\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt\" (UID: \"359c47fc-f8af-43f8-865b-1f187fb2672c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt"
Oct 30 19:28:26.715383 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.715343    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/359c47fc-f8af-43f8-865b-1f187fb2672c-istio-podinfo\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt\" (UID: \"359c47fc-f8af-43f8-865b-1f187fb2672c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt"
Oct 30 19:28:26.715566 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.715442    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/359c47fc-f8af-43f8-865b-1f187fb2672c-workload-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt\" (UID: \"359c47fc-f8af-43f8-865b-1f187fb2672c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt"
Oct 30 19:28:26.715566 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.715483    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/359c47fc-f8af-43f8-865b-1f187fb2672c-credential-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt\" (UID: \"359c47fc-f8af-43f8-865b-1f187fb2672c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt"
Oct 30 19:28:26.715566 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.715500    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-989kw\" (UniqueName: \"kubernetes.io/projected/359c47fc-f8af-43f8-865b-1f187fb2672c-kube-api-access-989kw\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt\" (UID: \"359c47fc-f8af-43f8-865b-1f187fb2672c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt"
Oct 30 19:28:26.715566 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.715518    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/359c47fc-f8af-43f8-865b-1f187fb2672c-workload-certs\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt\" (UID: \"359c47fc-f8af-43f8-865b-1f187fb2672c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt"
Oct 30 19:28:26.715836 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.715631    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/359c47fc-f8af-43f8-865b-1f187fb2672c-istio-envoy\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt\" (UID: \"359c47fc-f8af-43f8-865b-1f187fb2672c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt"
Oct 30 19:28:26.817012 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.816974    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f97dffca-f853-4390-8f4a-cbf04856f5e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-768af474-j5578\" (UID: \"f97dffca-f853-4390-8f4a-cbf04856f5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768af474-j5578"
Oct 30 19:28:26.817212 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.817109    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f97dffca-f853-4390-8f4a-cbf04856f5e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-768af474-j5578\" (UID: \"f97dffca-f853-4390-8f4a-cbf04856f5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768af474-j5578"
Oct 30 19:28:26.817212 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.817143    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/359c47fc-f8af-43f8-865b-1f187fb2672c-istiod-ca-cert\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt\" (UID: \"359c47fc-f8af-43f8-865b-1f187fb2672c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt"
Oct 30 19:28:26.817212 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.817165    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/359c47fc-f8af-43f8-865b-1f187fb2672c-istio-data\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt\" (UID: \"359c47fc-f8af-43f8-865b-1f187fb2672c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt"
Oct 30 19:28:26.817486 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.817233    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr7z4\" (UniqueName: \"kubernetes.io/projected/f97dffca-f853-4390-8f4a-cbf04856f5e4-kube-api-access-cr7z4\") pod \"maas-api-revoke-sub-768af474-j5578\" (UID: \"f97dffca-f853-4390-8f4a-cbf04856f5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768af474-j5578"
Oct 30 19:28:26.817486 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.817270    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/359c47fc-f8af-43f8-865b-1f187fb2672c-istio-token\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt\" (UID: \"359c47fc-f8af-43f8-865b-1f187fb2672c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt"
Oct 30 19:28:26.817486 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.817322    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/359c47fc-f8af-43f8-865b-1f187fb2672c-istio-podinfo\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt\" (UID: \"359c47fc-f8af-43f8-865b-1f187fb2672c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt"
Oct 30 19:28:26.817486 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.817413    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/359c47fc-f8af-43f8-865b-1f187fb2672c-workload-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt\" (UID: \"359c47fc-f8af-43f8-865b-1f187fb2672c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt"
Oct 30 19:28:26.817695 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.817482    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/359c47fc-f8af-43f8-865b-1f187fb2672c-credential-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt\" (UID: \"359c47fc-f8af-43f8-865b-1f187fb2672c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt"
Oct 30 19:28:26.817695 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.817539    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-989kw\" (UniqueName: \"kubernetes.io/projected/359c47fc-f8af-43f8-865b-1f187fb2672c-kube-api-access-989kw\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt\" (UID: \"359c47fc-f8af-43f8-865b-1f187fb2672c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt"
Oct 30 19:28:26.817695 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.817574    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/359c47fc-f8af-43f8-865b-1f187fb2672c-workload-certs\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt\" (UID: \"359c47fc-f8af-43f8-865b-1f187fb2672c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt"
Oct 30 19:28:26.817695 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.817541    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/359c47fc-f8af-43f8-865b-1f187fb2672c-istio-data\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt\" (UID: \"359c47fc-f8af-43f8-865b-1f187fb2672c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt"
Oct 30 19:28:26.817695 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.817666    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/359c47fc-f8af-43f8-865b-1f187fb2672c-istio-envoy\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt\" (UID: \"359c47fc-f8af-43f8-865b-1f187fb2672c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt"
Oct 30 19:28:26.817977 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.817778    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/359c47fc-f8af-43f8-865b-1f187fb2672c-workload-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt\" (UID: \"359c47fc-f8af-43f8-865b-1f187fb2672c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt"
Oct 30 19:28:26.817977 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.817942    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/359c47fc-f8af-43f8-865b-1f187fb2672c-istiod-ca-cert\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt\" (UID: \"359c47fc-f8af-43f8-865b-1f187fb2672c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt"
Oct 30 19:28:26.818130 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.818108    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/359c47fc-f8af-43f8-865b-1f187fb2672c-credential-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt\" (UID: \"359c47fc-f8af-43f8-865b-1f187fb2672c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt"
Oct 30 19:28:26.818195 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.818175    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/359c47fc-f8af-43f8-865b-1f187fb2672c-workload-certs\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt\" (UID: \"359c47fc-f8af-43f8-865b-1f187fb2672c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt"
Oct 30 19:28:26.820154 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.820135    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/359c47fc-f8af-43f8-865b-1f187fb2672c-istio-podinfo\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt\" (UID: \"359c47fc-f8af-43f8-865b-1f187fb2672c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt"
Oct 30 19:28:26.820324 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.820304    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/359c47fc-f8af-43f8-865b-1f187fb2672c-istio-envoy\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt\" (UID: \"359c47fc-f8af-43f8-865b-1f187fb2672c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt"
Oct 30 19:28:26.827166 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.827141    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-989kw\" (UniqueName: \"kubernetes.io/projected/359c47fc-f8af-43f8-865b-1f187fb2672c-kube-api-access-989kw\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt\" (UID: \"359c47fc-f8af-43f8-865b-1f187fb2672c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt"
Oct 30 19:28:26.827262 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.827235    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/359c47fc-f8af-43f8-865b-1f187fb2672c-istio-token\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt\" (UID: \"359c47fc-f8af-43f8-865b-1f187fb2672c\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt"
Oct 30 19:28:26.915293 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.915206    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt"
Oct 30 19:28:26.918840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.918802    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f97dffca-f853-4390-8f4a-cbf04856f5e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-768af474-j5578\" (UID: \"f97dffca-f853-4390-8f4a-cbf04856f5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768af474-j5578"
Oct 30 19:28:26.919000 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.918907    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f97dffca-f853-4390-8f4a-cbf04856f5e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-768af474-j5578\" (UID: \"f97dffca-f853-4390-8f4a-cbf04856f5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768af474-j5578"
Oct 30 19:28:26.919000 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.918966    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr7z4\" (UniqueName: \"kubernetes.io/projected/f97dffca-f853-4390-8f4a-cbf04856f5e4-kube-api-access-cr7z4\") pod \"maas-api-revoke-sub-768af474-j5578\" (UID: \"f97dffca-f853-4390-8f4a-cbf04856f5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768af474-j5578"
Oct 30 19:28:26.919849 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.919618    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f97dffca-f853-4390-8f4a-cbf04856f5e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-768af474-j5578\" (UID: \"f97dffca-f853-4390-8f4a-cbf04856f5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768af474-j5578"
Oct 30 19:28:26.921531 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.921506    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f97dffca-f853-4390-8f4a-cbf04856f5e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-768af474-j5578\" (UID: \"f97dffca-f853-4390-8f4a-cbf04856f5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768af474-j5578"
Oct 30 19:28:26.928308 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:26.928288    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr7z4\" (UniqueName: \"kubernetes.io/projected/f97dffca-f853-4390-8f4a-cbf04856f5e4-kube-api-access-cr7z4\") pod \"maas-api-revoke-sub-768af474-j5578\" (UID: \"f97dffca-f853-4390-8f4a-cbf04856f5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768af474-j5578"
Oct 30 19:28:27.009829 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:27.008961    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768af474-j5578"
Oct 30 19:28:27.190812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:27.190710    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t"]
Oct 30 19:28:27.197045 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:27.197016    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t"
Oct 30 19:28:27.200820 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:27.200791    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t"]
Oct 30 19:28:27.210543 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:27.210519    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt"]
Oct 30 19:28:27.211144 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:28:27.211122    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod359c47fc_f8af_43f8_865b_1f187fb2672c.slice/crio-5ede5bedc56af57330642fe3b9543a93e621fe7d4181c547b561aa8228402ca8 WatchSource:0}: Error finding container 5ede5bedc56af57330642fe3b9543a93e621fe7d4181c547b561aa8228402ca8: Status 404 returned error can't find the container with id 5ede5bedc56af57330642fe3b9543a93e621fe7d4181c547b561aa8228402ca8
Oct 30 19:28:27.214803 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:27.214764    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:28:27.214885 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:27.214852    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:28:27.214921 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:27.214893    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:28:27.324024 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:27.323992    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4897f8f5-579f-45ed-a964-f2379eb52848-istio-token\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t\" (UID: \"4897f8f5-579f-45ed-a964-f2379eb52848\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t"
Oct 30 19:28:27.324268 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:27.324243    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4897f8f5-579f-45ed-a964-f2379eb52848-istio-envoy\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t\" (UID: \"4897f8f5-579f-45ed-a964-f2379eb52848\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t"
Oct 30 19:28:27.324375 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:27.324346    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4897f8f5-579f-45ed-a964-f2379eb52848-workload-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t\" (UID: \"4897f8f5-579f-45ed-a964-f2379eb52848\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t"
Oct 30 19:28:27.324448 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:27.324371    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4897f8f5-579f-45ed-a964-f2379eb52848-istio-data\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t\" (UID: \"4897f8f5-579f-45ed-a964-f2379eb52848\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t"
Oct 30 19:28:27.324448 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:27.324409    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4897f8f5-579f-45ed-a964-f2379eb52848-credential-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t\" (UID: \"4897f8f5-579f-45ed-a964-f2379eb52848\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t"
Oct 30 19:28:27.324448 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:27.324433    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4897f8f5-579f-45ed-a964-f2379eb52848-workload-certs\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t\" (UID: \"4897f8f5-579f-45ed-a964-f2379eb52848\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t"
Oct 30 19:28:27.324607 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:27.324497    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4897f8f5-579f-45ed-a964-f2379eb52848-istiod-ca-cert\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t\" (UID: \"4897f8f5-579f-45ed-a964-f2379eb52848\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t"
Oct 30 19:28:27.324607 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:27.324533    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw2dp\" (UniqueName: \"kubernetes.io/projected/4897f8f5-579f-45ed-a964-f2379eb52848-kube-api-access-dw2dp\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t\" (UID: \"4897f8f5-579f-45ed-a964-f2379eb52848\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t"
Oct 30 19:28:27.324723 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:27.324610    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4897f8f5-579f-45ed-a964-f2379eb52848-istio-podinfo\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t\" (UID: \"4897f8f5-579f-45ed-a964-f2379eb52848\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t"
Oct 30 19:28:27.327172 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:28:27.326914    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf97dffca_f853_4390_8f4a_cbf04856f5e4.slice/crio-ebe6496082eaf9ce8bcda988904262da3ea2843f414a26f358d138171a4be594 WatchSource:0}: Error finding container ebe6496082eaf9ce8bcda988904262da3ea2843f414a26f358d138171a4be594: Status 404 returned error can't find the container with id ebe6496082eaf9ce8bcda988904262da3ea2843f414a26f358d138171a4be594
Oct 30 19:28:27.327384 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:27.327007    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-768af474-j5578"]
Oct 30 19:28:27.426252 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:27.426215    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4897f8f5-579f-45ed-a964-f2379eb52848-istiod-ca-cert\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t\" (UID: \"4897f8f5-579f-45ed-a964-f2379eb52848\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t"
Oct 30 19:28:27.426458 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:27.426279    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw2dp\" (UniqueName: \"kubernetes.io/projected/4897f8f5-579f-45ed-a964-f2379eb52848-kube-api-access-dw2dp\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t\" (UID: \"4897f8f5-579f-45ed-a964-f2379eb52848\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t"
Oct 30 19:28:27.426458 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:27.426339    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4897f8f5-579f-45ed-a964-f2379eb52848-istio-podinfo\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t\" (UID: \"4897f8f5-579f-45ed-a964-f2379eb52848\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t"
Oct 30 19:28:27.426572 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:27.426470    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4897f8f5-579f-45ed-a964-f2379eb52848-istio-token\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t\" (UID: \"4897f8f5-579f-45ed-a964-f2379eb52848\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t"
Oct 30 19:28:27.426572 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:27.426506    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4897f8f5-579f-45ed-a964-f2379eb52848-istio-envoy\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t\" (UID: \"4897f8f5-579f-45ed-a964-f2379eb52848\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t"
Oct 30 19:28:27.426670 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:27.426569    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4897f8f5-579f-45ed-a964-f2379eb52848-workload-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t\" (UID: \"4897f8f5-579f-45ed-a964-f2379eb52848\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t"
Oct 30 19:28:27.426670 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:27.426592    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4897f8f5-579f-45ed-a964-f2379eb52848-istio-data\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t\" (UID: \"4897f8f5-579f-45ed-a964-f2379eb52848\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t"
Oct 30 19:28:27.426670 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:27.426620    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4897f8f5-579f-45ed-a964-f2379eb52848-credential-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t\" (UID: \"4897f8f5-579f-45ed-a964-f2379eb52848\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t"
Oct 30 19:28:27.426670 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:27.426641    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4897f8f5-579f-45ed-a964-f2379eb52848-workload-certs\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t\" (UID: \"4897f8f5-579f-45ed-a964-f2379eb52848\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t"
Oct 30 19:28:27.427026 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:27.426994    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4897f8f5-579f-45ed-a964-f2379eb52848-istio-data\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t\" (UID: \"4897f8f5-579f-45ed-a964-f2379eb52848\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t"
Oct 30 19:28:27.427151 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:27.427101    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4897f8f5-579f-45ed-a964-f2379eb52848-workload-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t\" (UID: \"4897f8f5-579f-45ed-a964-f2379eb52848\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t"
Oct 30 19:28:27.427265 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:27.427242    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4897f8f5-579f-45ed-a964-f2379eb52848-istiod-ca-cert\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t\" (UID: \"4897f8f5-579f-45ed-a964-f2379eb52848\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t"
Oct 30 19:28:27.427265 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:27.427255    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4897f8f5-579f-45ed-a964-f2379eb52848-workload-certs\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t\" (UID: \"4897f8f5-579f-45ed-a964-f2379eb52848\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t"
Oct 30 19:28:27.427411 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:27.427295    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4897f8f5-579f-45ed-a964-f2379eb52848-credential-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t\" (UID: \"4897f8f5-579f-45ed-a964-f2379eb52848\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t"
Oct 30 19:28:27.429300 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:27.429276    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4897f8f5-579f-45ed-a964-f2379eb52848-istio-podinfo\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t\" (UID: \"4897f8f5-579f-45ed-a964-f2379eb52848\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t"
Oct 30 19:28:27.429422 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:27.429351    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4897f8f5-579f-45ed-a964-f2379eb52848-istio-envoy\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t\" (UID: \"4897f8f5-579f-45ed-a964-f2379eb52848\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t"
Oct 30 19:28:27.434530 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:27.434504    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4897f8f5-579f-45ed-a964-f2379eb52848-istio-token\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t\" (UID: \"4897f8f5-579f-45ed-a964-f2379eb52848\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t"
Oct 30 19:28:27.434657 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:27.434620    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw2dp\" (UniqueName: \"kubernetes.io/projected/4897f8f5-579f-45ed-a964-f2379eb52848-kube-api-access-dw2dp\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t\" (UID: \"4897f8f5-579f-45ed-a964-f2379eb52848\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t"
Oct 30 19:28:27.518685 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:27.518654    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t"
Oct 30 19:28:27.813247 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:27.813219    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t"]
Oct 30 19:28:27.813569 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:28:27.813225    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4897f8f5_579f_45ed_a964_f2379eb52848.slice/crio-34b9a8c03add878837b79d96240d8b7347b23f3164cec37e50bb8a9c34a47e8d WatchSource:0}: Error finding container 34b9a8c03add878837b79d96240d8b7347b23f3164cec37e50bb8a9c34a47e8d: Status 404 returned error can't find the container with id 34b9a8c03add878837b79d96240d8b7347b23f3164cec37e50bb8a9c34a47e8d
Oct 30 19:28:27.815806 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:27.815772    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:28:27.815925 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:27.815836    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:28:27.815925 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:27.815885    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:28:28.207963 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:28.207856    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768af474-j5578" event={"ID":"f97dffca-f853-4390-8f4a-cbf04856f5e4","Type":"ContainerStarted","Data":"f41e7fe084339020c51b51b0770b499bf29fcffa24e91e1c2acd320e22fb6e3c"}
Oct 30 19:28:28.208133 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:28.207967    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768af474-j5578" event={"ID":"f97dffca-f853-4390-8f4a-cbf04856f5e4","Type":"ContainerStarted","Data":"ebe6496082eaf9ce8bcda988904262da3ea2843f414a26f358d138171a4be594"}
Oct 30 19:28:28.210465 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:28.210435    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt" event={"ID":"359c47fc-f8af-43f8-865b-1f187fb2672c","Type":"ContainerStarted","Data":"2b329b14627cf06b423a2cbca22183d9d506012c6472357ef35919dc3b4171f4"}
Oct 30 19:28:28.210609 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:28.210489    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt" event={"ID":"359c47fc-f8af-43f8-865b-1f187fb2672c","Type":"ContainerStarted","Data":"5ede5bedc56af57330642fe3b9543a93e621fe7d4181c547b561aa8228402ca8"}
Oct 30 19:28:28.213049 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:28.212991    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t" event={"ID":"4897f8f5-579f-45ed-a964-f2379eb52848","Type":"ContainerStarted","Data":"ed9f07518d42efa6ff32cfd0e47b68eded8fba60afb242e22f90fcffff274993"}
Oct 30 19:28:28.213049 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:28.213031    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t" event={"ID":"4897f8f5-579f-45ed-a964-f2379eb52848","Type":"ContainerStarted","Data":"34b9a8c03add878837b79d96240d8b7347b23f3164cec37e50bb8a9c34a47e8d"}
Oct 30 19:28:28.227418 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:28.227360    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768af474-j5578" podStartSLOduration=2.227341135 podStartE2EDuration="2.227341135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:28.224671519 +0000 UTC m=+727.735731485" watchObservedRunningTime="2026-10-30 19:28:28.227341135 +0000 UTC m=+727.738401102"
Oct 30 19:28:28.251713 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:28.251651    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t" podStartSLOduration=1.251630116 podStartE2EDuration="1.251630116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:28.246807127 +0000 UTC m=+727.757867091" watchObservedRunningTime="2026-10-30 19:28:28.251630116 +0000 UTC m=+727.762690081"
Oct 30 19:28:28.519513 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:28.519476    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t"
Oct 30 19:28:28.521291 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:28.521271    5043 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 19:28:28.521347 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:28.521328    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t" podUID="4897f8f5-579f-45ed-a964-f2379eb52848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 19:28:28.917179 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:28.917089    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt"
Oct 30 19:28:28.920924 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:28.920891    5043 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 19:28:28.921420 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:28.921389    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt" podUID="359c47fc-f8af-43f8-865b-1f187fb2672c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 19:28:29.227971 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:29.227935    5043 generic.go:356] "Generic (PLEG): container finished" podID="f97dffca-f853-4390-8f4a-cbf04856f5e4" containerID="f41e7fe084339020c51b51b0770b499bf29fcffa24e91e1c2acd320e22fb6e3c" exitCode=0
Oct 30 19:28:29.228867 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:29.228843    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768af474-j5578" event={"ID":"f97dffca-f853-4390-8f4a-cbf04856f5e4","Type":"ContainerDied","Data":"f41e7fe084339020c51b51b0770b499bf29fcffa24e91e1c2acd320e22fb6e3c"}
Oct 30 19:28:29.250475 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:29.250422    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt" podStartSLOduration=3.25040812 podStartE2EDuration="3.25040812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:28.29383276 +0000 UTC m=+727.804892728" watchObservedRunningTime="2026-10-30 19:28:29.25040812 +0000 UTC m=+728.761468084"
Oct 30 19:28:29.519231 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:29.519134    5043 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 19:28:29.519505 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:29.519472    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t" podUID="4897f8f5-579f-45ed-a964-f2379eb52848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 19:28:29.916402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:29.916303    5043 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 19:28:29.917100 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:29.916658    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt" podUID="359c47fc-f8af-43f8-865b-1f187fb2672c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 19:28:30.524570 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:30.524188    5043 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 19:28:30.524570 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:30.524239    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t" podUID="4897f8f5-579f-45ed-a964-f2379eb52848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 19:28:30.726062 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:30.726021    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768af474-j5578"
Oct 30 19:28:30.829983 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:30.829785    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f97dffca-f853-4390-8f4a-cbf04856f5e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f97dffca-f853-4390-8f4a-cbf04856f5e4-maas-api-xks-ca\") pod \"f97dffca-f853-4390-8f4a-cbf04856f5e4\" (UID: \"f97dffca-f853-4390-8f4a-cbf04856f5e4\") "
Oct 30 19:28:30.830647 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:30.830623    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f97dffca-f853-4390-8f4a-cbf04856f5e4-kube-api-access-cr7z4\" (UniqueName: \"kubernetes.io/projected/f97dffca-f853-4390-8f4a-cbf04856f5e4-kube-api-access-cr7z4\") pod \"f97dffca-f853-4390-8f4a-cbf04856f5e4\" (UID: \"f97dffca-f853-4390-8f4a-cbf04856f5e4\") "
Oct 30 19:28:30.830985 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:30.830966    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f97dffca-f853-4390-8f4a-cbf04856f5e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f97dffca-f853-4390-8f4a-cbf04856f5e4-maas-api-openshift-ca\") pod \"f97dffca-f853-4390-8f4a-cbf04856f5e4\" (UID: \"f97dffca-f853-4390-8f4a-cbf04856f5e4\") "
Oct 30 19:28:30.832921 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:30.831603    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f97dffca-f853-4390-8f4a-cbf04856f5e4-maas-api-openshift-ca" pod "f97dffca-f853-4390-8f4a-cbf04856f5e4" (UID: "f97dffca-f853-4390-8f4a-cbf04856f5e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:28:30.833431 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:30.833410    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f97dffca-f853-4390-8f4a-cbf04856f5e4-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:30.837629 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:30.837590    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f97dffca-f853-4390-8f4a-cbf04856f5e4-maas-api-xks-ca" pod "f97dffca-f853-4390-8f4a-cbf04856f5e4" (UID: "f97dffca-f853-4390-8f4a-cbf04856f5e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:28:30.842272 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:30.842242    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f97dffca-f853-4390-8f4a-cbf04856f5e4-kube-api-access-cr7z4" pod "f97dffca-f853-4390-8f4a-cbf04856f5e4" (UID: "f97dffca-f853-4390-8f4a-cbf04856f5e4"). InnerVolumeSpecName "kube-api-access-cr7z4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:28:30.928712 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:30.928676    5043 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 19:28:30.931359 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:30.930835    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt" podUID="359c47fc-f8af-43f8-865b-1f187fb2672c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 19:28:30.935452 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:30.935421    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f97dffca-f853-4390-8f4a-cbf04856f5e4-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:30.935600 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:30.935455    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr7z4\" (UniqueName: \"kubernetes.io/projected/f97dffca-f853-4390-8f4a-cbf04856f5e4-kube-api-access-cr7z4\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:31.244762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:31.244555    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768af474-j5578" event={"ID":"f97dffca-f853-4390-8f4a-cbf04856f5e4","Type":"ContainerDied","Data":"ebe6496082eaf9ce8bcda988904262da3ea2843f414a26f358d138171a4be594"}
Oct 30 19:28:31.244762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:31.244603    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebe6496082eaf9ce8bcda988904262da3ea2843f414a26f358d138171a4be594"
Oct 30 19:28:31.244762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:31.244655    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768af474-j5578"
Oct 30 19:28:31.526824 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:31.526202    5043 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 19:28:31.526824 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:31.526295    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t" podUID="4897f8f5-579f-45ed-a964-f2379eb52848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 19:28:31.916247 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:31.916151    5043 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 19:28:31.916499 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:31.916471    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt" podUID="359c47fc-f8af-43f8-865b-1f187fb2672c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 19:28:32.520763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:32.520122    5043 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 19:28:32.520763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:32.520184    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t" podUID="4897f8f5-579f-45ed-a964-f2379eb52848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 19:28:32.916198 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:32.916106    5043 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 19:28:32.916802 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:32.916769    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt" podUID="359c47fc-f8af-43f8-865b-1f187fb2672c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 19:28:32.949868 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:32.949826    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:33.294763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:33.292912    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:33.507879 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:33.497995    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:33.524329 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:33.523902    5043 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 19:28:33.524329 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:33.523986    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t" podUID="4897f8f5-579f-45ed-a964-f2379eb52848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 19:28:33.604858 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:33.604566    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:33.734799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:33.733837    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:33.772775 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:33.770837    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:33.834763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:33.833897    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:33.876822 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:33.871838    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:33.937227 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:33.937185    5043 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 19:28:33.937906 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:33.937857    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt" podUID="359c47fc-f8af-43f8-865b-1f187fb2672c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 19:28:33.953805 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:33.953760    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:33.955453 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:33.955429    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:28:34.308141 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:34.290882    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:28:34.308141 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:34.295983    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:28:34.406774 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:34.406581    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:28:34.540082 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:34.539546    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:28:34.558826 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:34.541567    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:28:34.585178 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:34.569908    5043 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 19:28:34.652762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:34.629001    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t" podUID="4897f8f5-579f-45ed-a964-f2379eb52848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 19:28:34.684694 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:34.684653    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:28:34.734629 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:34.734584    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:34.853821 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:34.842030    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:34.872763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:34.861126    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:34.873915 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:34.873874    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:34.936769 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:34.915889    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:28:34.936769 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:34.915957    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:28:34.936769 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:34.915991    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:28:35.182199 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:35.182106    5043 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 19:28:35.211859 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:35.211810    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt" podUID="359c47fc-f8af-43f8-865b-1f187fb2672c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 19:28:35.289529 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:35.289478    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:35.323798 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:35.310868    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:28:35.523148 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:35.523099    5043 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 19:28:35.523472 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:35.523421    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t" podUID="4897f8f5-579f-45ed-a964-f2379eb52848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 19:28:35.701771 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:35.687988    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:35.720759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:35.720686    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:35.723924 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:35.723893    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:28:35.749879 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:35.749851    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:28:35.915968 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:35.915854    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:35.925768 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:35.916465    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:28:35.963987 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:35.956430    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:28:35.963987 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:35.957181    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:28:35.974953 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:35.974915    5043 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 19:28:35.975237 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:35.975208    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt" podUID="359c47fc-f8af-43f8-865b-1f187fb2672c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 19:28:35.976116 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:35.976095    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:28:36.046634 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:36.046553    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"]
Oct 30 19:28:36.050491 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:36.050462    5043 cpu_manager.go:414] "containerMap: removing container" podUID="f97dffca-f853-4390-8f4a-cbf04856f5e4" containerName="revoke-keys"
Oct 30 19:28:36.050691 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:36.050676    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f97dffca-f853-4390-8f4a-cbf04856f5e4" containerName="revoke-keys"
Oct 30 19:28:36.051057 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:36.051039    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="f97dffca-f853-4390-8f4a-cbf04856f5e4" containerName="revoke-keys"
Oct 30 19:28:36.054718 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:36.054698    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:28:36.055466 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:36.055448    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"
Oct 30 19:28:36.079018 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:36.078989    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-01b24e\"/\"distinct-2-w6-01b24e-kserve-self-signed-certs\""
Oct 30 19:28:36.220548 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:36.084523    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:28:36.232445 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:36.232416    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"]
Oct 30 19:28:36.232750 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:36.232720    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:28:36.232880 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:36.157947    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71b8006f-09c2-4f6a-a315-3687f72aed60-tmp-dir\") pod \"distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5\" (UID: \"71b8006f-09c2-4f6a-a315-3687f72aed60\") " pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"
Oct 30 19:28:36.233064 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:36.233051    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71b8006f-09c2-4f6a-a315-3687f72aed60-model-cache\") pod \"distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5\" (UID: \"71b8006f-09c2-4f6a-a315-3687f72aed60\") " pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"
Oct 30 19:28:36.233189 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:36.233178    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71b8006f-09c2-4f6a-a315-3687f72aed60-dshm\") pod \"distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5\" (UID: \"71b8006f-09c2-4f6a-a315-3687f72aed60\") " pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"
Oct 30 19:28:36.233363 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:36.233349    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71b8006f-09c2-4f6a-a315-3687f72aed60-tls-certs\") pod \"distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5\" (UID: \"71b8006f-09c2-4f6a-a315-3687f72aed60\") " pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"
Oct 30 19:28:36.233531 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:36.233516    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71b8006f-09c2-4f6a-a315-3687f72aed60-home\") pod \"distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5\" (UID: \"71b8006f-09c2-4f6a-a315-3687f72aed60\") " pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"
Oct 30 19:28:36.233666 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:36.233652    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrv2t\" (UniqueName: \"kubernetes.io/projected/71b8006f-09c2-4f6a-a315-3687f72aed60-kube-api-access-vrv2t\") pod \"distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5\" (UID: \"71b8006f-09c2-4f6a-a315-3687f72aed60\") " pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"
Oct 30 19:28:36.249578 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:36.249551    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:28:36.285063 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:36.261727    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:28:36.293895 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:36.293069    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:28:36.335729 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:36.335689    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71b8006f-09c2-4f6a-a315-3687f72aed60-model-cache\") pod \"distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5\" (UID: \"71b8006f-09c2-4f6a-a315-3687f72aed60\") " pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"
Oct 30 19:28:36.336132 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:36.336102    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71b8006f-09c2-4f6a-a315-3687f72aed60-dshm\") pod \"distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5\" (UID: \"71b8006f-09c2-4f6a-a315-3687f72aed60\") " pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"
Oct 30 19:28:36.336233 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:36.336194    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71b8006f-09c2-4f6a-a315-3687f72aed60-tls-certs\") pod \"distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5\" (UID: \"71b8006f-09c2-4f6a-a315-3687f72aed60\") " pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"
Oct 30 19:28:36.336316 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:36.336299    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71b8006f-09c2-4f6a-a315-3687f72aed60-home\") pod \"distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5\" (UID: \"71b8006f-09c2-4f6a-a315-3687f72aed60\") " pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"
Oct 30 19:28:36.336403 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:36.336386    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrv2t\" (UniqueName: \"kubernetes.io/projected/71b8006f-09c2-4f6a-a315-3687f72aed60-kube-api-access-vrv2t\") pod \"distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5\" (UID: \"71b8006f-09c2-4f6a-a315-3687f72aed60\") " pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"
Oct 30 19:28:36.336570 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:36.336554    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71b8006f-09c2-4f6a-a315-3687f72aed60-tmp-dir\") pod \"distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5\" (UID: \"71b8006f-09c2-4f6a-a315-3687f72aed60\") " pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"
Oct 30 19:28:36.337054 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:36.337026    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71b8006f-09c2-4f6a-a315-3687f72aed60-tmp-dir\") pod \"distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5\" (UID: \"71b8006f-09c2-4f6a-a315-3687f72aed60\") " pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"
Oct 30 19:28:36.337913 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:36.337887    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71b8006f-09c2-4f6a-a315-3687f72aed60-model-cache\") pod \"distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5\" (UID: \"71b8006f-09c2-4f6a-a315-3687f72aed60\") " pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"
Oct 30 19:28:36.340414 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:36.340389    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71b8006f-09c2-4f6a-a315-3687f72aed60-home\") pod \"distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5\" (UID: \"71b8006f-09c2-4f6a-a315-3687f72aed60\") " pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"
Oct 30 19:28:36.343783 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:36.343758    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71b8006f-09c2-4f6a-a315-3687f72aed60-dshm\") pod \"distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5\" (UID: \"71b8006f-09c2-4f6a-a315-3687f72aed60\") " pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"
Oct 30 19:28:36.345834 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:36.345810    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71b8006f-09c2-4f6a-a315-3687f72aed60-tls-certs\") pod \"distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5\" (UID: \"71b8006f-09c2-4f6a-a315-3687f72aed60\") " pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"
Oct 30 19:28:36.367207 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:36.367175    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrv2t\" (UniqueName: \"kubernetes.io/projected/71b8006f-09c2-4f6a-a315-3687f72aed60-kube-api-access-vrv2t\") pod \"distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5\" (UID: \"71b8006f-09c2-4f6a-a315-3687f72aed60\") " pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"
Oct 30 19:28:36.527279 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:36.527181    5043 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 19:28:36.528601 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:36.528568    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t" podUID="4897f8f5-579f-45ed-a964-f2379eb52848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 19:28:36.567206 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:36.567167    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"
Oct 30 19:28:37.262758 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:37.261869    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"]
Oct 30 19:28:37.284757 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:28:37.281860    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71b8006f_09c2_4f6a_a315_3687f72aed60.slice/crio-58b4ef32b28df501b45992c8bef25c08bb984c599bdcc4ed463d039bfdd9d71c WatchSource:0}: Error finding container 58b4ef32b28df501b45992c8bef25c08bb984c599bdcc4ed463d039bfdd9d71c: Status 404 returned error can't find the container with id 58b4ef32b28df501b45992c8bef25c08bb984c599bdcc4ed463d039bfdd9d71c
Oct 30 19:28:37.964445 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:37.963842    5043 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:28:37.964445 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:37.963910    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt" podUID="359c47fc-f8af-43f8-865b-1f187fb2672c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:38.140853 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:38.140812    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" event={"ID":"71b8006f-09c2-4f6a-a315-3687f72aed60","Type":"ContainerStarted","Data":"7996187c5b2d2513fb75dc201cc1aebb27aa9cf8db792fc0839d36a01ca852b7"}
Oct 30 19:28:38.141022 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:38.140884    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" event={"ID":"71b8006f-09c2-4f6a-a315-3687f72aed60","Type":"ContainerStarted","Data":"58b4ef32b28df501b45992c8bef25c08bb984c599bdcc4ed463d039bfdd9d71c"}
Oct 30 19:28:38.181413 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:38.176769    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podStartSLOduration=3.176747103 podStartE2EDuration="3.176747103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:38.162871004 +0000 UTC m=+737.673930970" watchObservedRunningTime="2026-10-30 19:28:38.176747103 +0000 UTC m=+737.687807062"
Oct 30 19:28:38.417333 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:38.417237    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t"
Oct 30 19:28:38.418267 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:38.418229    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t"
Oct 30 19:28:38.419198 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:38.419180    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t"
Oct 30 19:28:38.444773 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:38.444705    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt"
Oct 30 19:28:38.445870 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:38.445693    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt"
Oct 30 19:28:38.448144 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:38.448124    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt"
Oct 30 19:28:38.568359 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:38.568331    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"
Oct 30 19:28:41.385479 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:41.385445    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg"]
Oct 30 19:28:41.390454 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:41.390424    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg"
Oct 30 19:28:41.465725 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:41.465686    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a851569d-8aa4-4af2-9f3c-c90c4895b95a-workload-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg\" (UID: \"a851569d-8aa4-4af2-9f3c-c90c4895b95a\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg"
Oct 30 19:28:41.465725 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:41.465727    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a851569d-8aa4-4af2-9f3c-c90c4895b95a-istiod-ca-cert\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg\" (UID: \"a851569d-8aa4-4af2-9f3c-c90c4895b95a\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg"
Oct 30 19:28:41.465971 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:41.465864    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a851569d-8aa4-4af2-9f3c-c90c4895b95a-istio-token\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg\" (UID: \"a851569d-8aa4-4af2-9f3c-c90c4895b95a\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg"
Oct 30 19:28:41.465971 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:41.465891    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a851569d-8aa4-4af2-9f3c-c90c4895b95a-istio-podinfo\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg\" (UID: \"a851569d-8aa4-4af2-9f3c-c90c4895b95a\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg"
Oct 30 19:28:41.466101 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:41.466000    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a851569d-8aa4-4af2-9f3c-c90c4895b95a-workload-certs\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg\" (UID: \"a851569d-8aa4-4af2-9f3c-c90c4895b95a\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg"
Oct 30 19:28:41.466210 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:41.466194    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h89m8\" (UniqueName: \"kubernetes.io/projected/a851569d-8aa4-4af2-9f3c-c90c4895b95a-kube-api-access-h89m8\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg\" (UID: \"a851569d-8aa4-4af2-9f3c-c90c4895b95a\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg"
Oct 30 19:28:41.466271 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:41.466232    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a851569d-8aa4-4af2-9f3c-c90c4895b95a-credential-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg\" (UID: \"a851569d-8aa4-4af2-9f3c-c90c4895b95a\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg"
Oct 30 19:28:41.466271 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:41.466264    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a851569d-8aa4-4af2-9f3c-c90c4895b95a-istio-envoy\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg\" (UID: \"a851569d-8aa4-4af2-9f3c-c90c4895b95a\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg"
Oct 30 19:28:41.466343 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:41.466326    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a851569d-8aa4-4af2-9f3c-c90c4895b95a-istio-data\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg\" (UID: \"a851569d-8aa4-4af2-9f3c-c90c4895b95a\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg"
Oct 30 19:28:41.509956 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:41.509923    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg"]
Oct 30 19:28:41.567599 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:41.567562    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a851569d-8aa4-4af2-9f3c-c90c4895b95a-workload-certs\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg\" (UID: \"a851569d-8aa4-4af2-9f3c-c90c4895b95a\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg"
Oct 30 19:28:41.567777 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:41.567697    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h89m8\" (UniqueName: \"kubernetes.io/projected/a851569d-8aa4-4af2-9f3c-c90c4895b95a-kube-api-access-h89m8\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg\" (UID: \"a851569d-8aa4-4af2-9f3c-c90c4895b95a\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg"
Oct 30 19:28:41.567777 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:41.567725    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a851569d-8aa4-4af2-9f3c-c90c4895b95a-credential-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg\" (UID: \"a851569d-8aa4-4af2-9f3c-c90c4895b95a\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg"
Oct 30 19:28:41.567777 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:41.567768    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a851569d-8aa4-4af2-9f3c-c90c4895b95a-istio-envoy\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg\" (UID: \"a851569d-8aa4-4af2-9f3c-c90c4895b95a\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg"
Oct 30 19:28:41.567965 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:41.567806    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a851569d-8aa4-4af2-9f3c-c90c4895b95a-istio-data\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg\" (UID: \"a851569d-8aa4-4af2-9f3c-c90c4895b95a\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg"
Oct 30 19:28:41.567965 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:41.567851    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a851569d-8aa4-4af2-9f3c-c90c4895b95a-workload-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg\" (UID: \"a851569d-8aa4-4af2-9f3c-c90c4895b95a\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg"
Oct 30 19:28:41.567965 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:41.567869    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a851569d-8aa4-4af2-9f3c-c90c4895b95a-istiod-ca-cert\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg\" (UID: \"a851569d-8aa4-4af2-9f3c-c90c4895b95a\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg"
Oct 30 19:28:41.567965 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:41.567947    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a851569d-8aa4-4af2-9f3c-c90c4895b95a-workload-certs\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg\" (UID: \"a851569d-8aa4-4af2-9f3c-c90c4895b95a\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg"
Oct 30 19:28:41.568177 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:41.568009    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a851569d-8aa4-4af2-9f3c-c90c4895b95a-istio-token\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg\" (UID: \"a851569d-8aa4-4af2-9f3c-c90c4895b95a\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg"
Oct 30 19:28:41.568177 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:41.568039    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a851569d-8aa4-4af2-9f3c-c90c4895b95a-istio-podinfo\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg\" (UID: \"a851569d-8aa4-4af2-9f3c-c90c4895b95a\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg"
Oct 30 19:28:41.568177 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:41.568095    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a851569d-8aa4-4af2-9f3c-c90c4895b95a-credential-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg\" (UID: \"a851569d-8aa4-4af2-9f3c-c90c4895b95a\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg"
Oct 30 19:28:41.568452 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:41.568425    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a851569d-8aa4-4af2-9f3c-c90c4895b95a-istio-data\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg\" (UID: \"a851569d-8aa4-4af2-9f3c-c90c4895b95a\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg"
Oct 30 19:28:41.568541 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:41.568467    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a851569d-8aa4-4af2-9f3c-c90c4895b95a-workload-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg\" (UID: \"a851569d-8aa4-4af2-9f3c-c90c4895b95a\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg"
Oct 30 19:28:41.568541 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:41.568470    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a851569d-8aa4-4af2-9f3c-c90c4895b95a-istiod-ca-cert\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg\" (UID: \"a851569d-8aa4-4af2-9f3c-c90c4895b95a\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg"
Oct 30 19:28:41.570421 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:41.570395    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a851569d-8aa4-4af2-9f3c-c90c4895b95a-istio-envoy\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg\" (UID: \"a851569d-8aa4-4af2-9f3c-c90c4895b95a\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg"
Oct 30 19:28:41.570977 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:41.570956    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a851569d-8aa4-4af2-9f3c-c90c4895b95a-istio-podinfo\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg\" (UID: \"a851569d-8aa4-4af2-9f3c-c90c4895b95a\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg"
Oct 30 19:28:41.593892 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:41.593855    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a851569d-8aa4-4af2-9f3c-c90c4895b95a-istio-token\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg\" (UID: \"a851569d-8aa4-4af2-9f3c-c90c4895b95a\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg"
Oct 30 19:28:41.594033 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:41.594012    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h89m8\" (UniqueName: \"kubernetes.io/projected/a851569d-8aa4-4af2-9f3c-c90c4895b95a-kube-api-access-h89m8\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg\" (UID: \"a851569d-8aa4-4af2-9f3c-c90c4895b95a\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg"
Oct 30 19:28:41.706721 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:41.706636    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg"
Oct 30 19:28:42.017990 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:42.017958    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg"]
Oct 30 19:28:42.024006 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:28:42.023960    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda851569d_8aa4_4af2_9f3c_c90c4895b95a.slice/crio-693b2f6b8ec1a093329a6ba6da4226c1bdf082a8018f608599260933a86a1527 WatchSource:0}: Error finding container 693b2f6b8ec1a093329a6ba6da4226c1bdf082a8018f608599260933a86a1527: Status 404 returned error can't find the container with id 693b2f6b8ec1a093329a6ba6da4226c1bdf082a8018f608599260933a86a1527
Oct 30 19:28:42.027939 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:42.027898    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:28:42.028048 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:42.027993    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:28:42.028048 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:42.028038    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:28:42.177972 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:42.177932    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg" event={"ID":"a851569d-8aa4-4af2-9f3c-c90c4895b95a","Type":"ContainerStarted","Data":"a6b31cdbbfe4e45387fe8fd60975db8818bece18976ce5c683d46c3fd9affd72"}
Oct 30 19:28:42.178187 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:42.177988    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg" event={"ID":"a851569d-8aa4-4af2-9f3c-c90c4895b95a","Type":"ContainerStarted","Data":"693b2f6b8ec1a093329a6ba6da4226c1bdf082a8018f608599260933a86a1527"}
Oct 30 19:28:42.222613 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:42.221252    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg" podStartSLOduration=1.221226003 podStartE2EDuration="1.221226003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:42.199155029 +0000 UTC m=+741.710215018" watchObservedRunningTime="2026-10-30 19:28:42.221226003 +0000 UTC m=+741.732285968"
Oct 30 19:28:42.707028 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:42.706986    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg"
Oct 30 19:28:43.708688 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:43.708658    5043 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 19:28:43.709085 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:43.708704    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg" podUID="a851569d-8aa4-4af2-9f3c-c90c4895b95a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 19:28:44.707299 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:44.707258    5043 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 19:28:44.707517 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:44.707323    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg" podUID="a851569d-8aa4-4af2-9f3c-c90c4895b95a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 19:28:45.707284 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:45.707246    5043 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 19:28:45.707691 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:45.707303    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg" podUID="a851569d-8aa4-4af2-9f3c-c90c4895b95a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 19:28:46.573763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:46.571197    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"
Oct 30 19:28:46.599889 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:46.599854    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"
Oct 30 19:28:46.707398 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:46.707353    5043 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 19:28:46.707840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:46.707420    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg" podUID="a851569d-8aa4-4af2-9f3c-c90c4895b95a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 19:28:47.253094 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:47.253065    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"
Oct 30 19:28:47.707673 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:47.707580    5043 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 19:28:47.708273 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:47.708239    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg" podUID="a851569d-8aa4-4af2-9f3c-c90c4895b95a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 19:28:48.707865 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:48.707822    5043 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 30 19:28:48.708313 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:48.707914    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg" podUID="a851569d-8aa4-4af2-9f3c-c90c4895b95a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 30 19:28:49.712840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:49.712801    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg"
Oct 30 19:28:49.713325 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:49.713286    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg"
Oct 30 19:28:49.714133 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:49.714112    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg"
Oct 30 19:28:51.729727 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:51.729692    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"]
Oct 30 19:28:51.736145 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:51.736123    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"
Oct 30 19:28:51.744295 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:51.744264    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ae63bf\"/\"distinct-w0-ae63bf-kserve-self-signed-certs\""
Oct 30 19:28:51.745922 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:51.745896    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"]
Oct 30 19:28:51.833831 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:51.833803    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8076c40a-54a0-41a2-b1d6-0d0b13c52351-tls-certs\") pod \"distinct-w0-ae63bf-kserve-769676c696-nszz4\" (UID: \"8076c40a-54a0-41a2-b1d6-0d0b13c52351\") " pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"
Oct 30 19:28:51.833981 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:51.833856    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8076c40a-54a0-41a2-b1d6-0d0b13c52351-tmp-dir\") pod \"distinct-w0-ae63bf-kserve-769676c696-nszz4\" (UID: \"8076c40a-54a0-41a2-b1d6-0d0b13c52351\") " pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"
Oct 30 19:28:51.833981 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:51.833960    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8076c40a-54a0-41a2-b1d6-0d0b13c52351-model-cache\") pod \"distinct-w0-ae63bf-kserve-769676c696-nszz4\" (UID: \"8076c40a-54a0-41a2-b1d6-0d0b13c52351\") " pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"
Oct 30 19:28:51.834081 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:51.834022    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpw9f\" (UniqueName: \"kubernetes.io/projected/8076c40a-54a0-41a2-b1d6-0d0b13c52351-kube-api-access-zpw9f\") pod \"distinct-w0-ae63bf-kserve-769676c696-nszz4\" (UID: \"8076c40a-54a0-41a2-b1d6-0d0b13c52351\") " pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"
Oct 30 19:28:51.834128 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:51.834115    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8076c40a-54a0-41a2-b1d6-0d0b13c52351-dshm\") pod \"distinct-w0-ae63bf-kserve-769676c696-nszz4\" (UID: \"8076c40a-54a0-41a2-b1d6-0d0b13c52351\") " pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"
Oct 30 19:28:51.834183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:51.834172    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8076c40a-54a0-41a2-b1d6-0d0b13c52351-home\") pod \"distinct-w0-ae63bf-kserve-769676c696-nszz4\" (UID: \"8076c40a-54a0-41a2-b1d6-0d0b13c52351\") " pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"
Oct 30 19:28:51.936126 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:51.936095    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpw9f\" (UniqueName: \"kubernetes.io/projected/8076c40a-54a0-41a2-b1d6-0d0b13c52351-kube-api-access-zpw9f\") pod \"distinct-w0-ae63bf-kserve-769676c696-nszz4\" (UID: \"8076c40a-54a0-41a2-b1d6-0d0b13c52351\") " pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"
Oct 30 19:28:51.936308 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:51.936157    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8076c40a-54a0-41a2-b1d6-0d0b13c52351-dshm\") pod \"distinct-w0-ae63bf-kserve-769676c696-nszz4\" (UID: \"8076c40a-54a0-41a2-b1d6-0d0b13c52351\") " pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"
Oct 30 19:28:51.936308 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:51.936220    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8076c40a-54a0-41a2-b1d6-0d0b13c52351-home\") pod \"distinct-w0-ae63bf-kserve-769676c696-nszz4\" (UID: \"8076c40a-54a0-41a2-b1d6-0d0b13c52351\") " pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"
Oct 30 19:28:51.936413 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:51.936318    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8076c40a-54a0-41a2-b1d6-0d0b13c52351-tls-certs\") pod \"distinct-w0-ae63bf-kserve-769676c696-nszz4\" (UID: \"8076c40a-54a0-41a2-b1d6-0d0b13c52351\") " pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"
Oct 30 19:28:51.936413 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:51.936380    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8076c40a-54a0-41a2-b1d6-0d0b13c52351-tmp-dir\") pod \"distinct-w0-ae63bf-kserve-769676c696-nszz4\" (UID: \"8076c40a-54a0-41a2-b1d6-0d0b13c52351\") " pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"
Oct 30 19:28:51.936525 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:51.936435    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8076c40a-54a0-41a2-b1d6-0d0b13c52351-model-cache\") pod \"distinct-w0-ae63bf-kserve-769676c696-nszz4\" (UID: \"8076c40a-54a0-41a2-b1d6-0d0b13c52351\") " pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"
Oct 30 19:28:51.936783 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:51.936716    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8076c40a-54a0-41a2-b1d6-0d0b13c52351-home\") pod \"distinct-w0-ae63bf-kserve-769676c696-nszz4\" (UID: \"8076c40a-54a0-41a2-b1d6-0d0b13c52351\") " pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"
Oct 30 19:28:51.936963 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:51.936818    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8076c40a-54a0-41a2-b1d6-0d0b13c52351-tmp-dir\") pod \"distinct-w0-ae63bf-kserve-769676c696-nszz4\" (UID: \"8076c40a-54a0-41a2-b1d6-0d0b13c52351\") " pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"
Oct 30 19:28:51.936963 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:51.936940    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8076c40a-54a0-41a2-b1d6-0d0b13c52351-model-cache\") pod \"distinct-w0-ae63bf-kserve-769676c696-nszz4\" (UID: \"8076c40a-54a0-41a2-b1d6-0d0b13c52351\") " pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"
Oct 30 19:28:51.938646 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:51.938629    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8076c40a-54a0-41a2-b1d6-0d0b13c52351-dshm\") pod \"distinct-w0-ae63bf-kserve-769676c696-nszz4\" (UID: \"8076c40a-54a0-41a2-b1d6-0d0b13c52351\") " pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"
Oct 30 19:28:51.939121 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:51.939100    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8076c40a-54a0-41a2-b1d6-0d0b13c52351-tls-certs\") pod \"distinct-w0-ae63bf-kserve-769676c696-nszz4\" (UID: \"8076c40a-54a0-41a2-b1d6-0d0b13c52351\") " pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"
Oct 30 19:28:51.944268 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:51.944249    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpw9f\" (UniqueName: \"kubernetes.io/projected/8076c40a-54a0-41a2-b1d6-0d0b13c52351-kube-api-access-zpw9f\") pod \"distinct-w0-ae63bf-kserve-769676c696-nszz4\" (UID: \"8076c40a-54a0-41a2-b1d6-0d0b13c52351\") " pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"
Oct 30 19:28:52.053607 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:52.053525    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"
Oct 30 19:28:52.358051 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:52.358016    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"]
Oct 30 19:28:52.358375 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:28:52.358349    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8076c40a_54a0_41a2_b1d6_0d0b13c52351.slice/crio-9808062374c1814473ec017fddcd97df2aca89716c8415446aebaaee7ec9f2fa WatchSource:0}: Error finding container 9808062374c1814473ec017fddcd97df2aca89716c8415446aebaaee7ec9f2fa: Status 404 returned error can't find the container with id 9808062374c1814473ec017fddcd97df2aca89716c8415446aebaaee7ec9f2fa
Oct 30 19:28:53.294022 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:53.293985    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" event={"ID":"8076c40a-54a0-41a2-b1d6-0d0b13c52351","Type":"ContainerStarted","Data":"e748252e2dd735b0c75008dc5053528813b53df641e82f864c4785fa027f86d9"}
Oct 30 19:28:53.294548 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:53.294035    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" event={"ID":"8076c40a-54a0-41a2-b1d6-0d0b13c52351","Type":"ContainerStarted","Data":"9808062374c1814473ec017fddcd97df2aca89716c8415446aebaaee7ec9f2fa"}
Oct 30 19:28:53.314170 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:53.314122    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podStartSLOduration=2.314107745 podStartE2EDuration="2.314107745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:53.313783129 +0000 UTC m=+752.824843092" watchObservedRunningTime="2026-10-30 19:28:53.314107745 +0000 UTC m=+752.825167710"
Oct 30 19:28:53.824780 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:53.824703    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02944da9-v956g"]
Oct 30 19:28:53.829306 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:53.829285    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02944da9-v956g"
Oct 30 19:28:53.840777 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:53.838425    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:28:53.840777 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:53.838516    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:28:53.840777 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:53.838561    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:28:53.840777 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:53.838679    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:28:53.841120 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:53.841052    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02944da9-v956g"]
Oct 30 19:28:53.904208 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:53.902598    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zt46p"]
Oct 30 19:28:53.907923 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:53.907892    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zt46p"
Oct 30 19:28:53.915029 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:53.915001    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zt46p"]
Oct 30 19:28:53.939017 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:53.937960    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 19:28:53.939017 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:53.938043    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 19:28:53.939017 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:53.937960    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 19:28:53.939017 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:53.938078    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-h9bmh\""
Oct 30 19:28:53.964677 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:53.964642    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92500b73-8bdb-4bd9-8770-2a1247803adb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02944da9-v956g\" (UID: \"92500b73-8bdb-4bd9-8770-2a1247803adb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02944da9-v956g"
Oct 30 19:28:53.964874 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:53.964698    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqsln\" (UniqueName: \"kubernetes.io/projected/92500b73-8bdb-4bd9-8770-2a1247803adb-kube-api-access-cqsln\") pod \"maas-api-revoke-sub-02944da9-v956g\" (UID: \"92500b73-8bdb-4bd9-8770-2a1247803adb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02944da9-v956g"
Oct 30 19:28:53.964935 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:53.964870    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92500b73-8bdb-4bd9-8770-2a1247803adb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02944da9-v956g\" (UID: \"92500b73-8bdb-4bd9-8770-2a1247803adb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02944da9-v956g"
Oct 30 19:28:54.054000 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:54.053969    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"
Oct 30 19:28:54.066410 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:54.066373    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f15d5348-e5a7-44d7-9ba0-e1f58482ac21-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zt46p\" (UID: \"f15d5348-e5a7-44d7-9ba0-e1f58482ac21\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zt46p"
Oct 30 19:28:54.066410 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:54.066419    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f15d5348-e5a7-44d7-9ba0-e1f58482ac21-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zt46p\" (UID: \"f15d5348-e5a7-44d7-9ba0-e1f58482ac21\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zt46p"
Oct 30 19:28:54.066693 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:54.066447    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f15d5348-e5a7-44d7-9ba0-e1f58482ac21-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zt46p\" (UID: \"f15d5348-e5a7-44d7-9ba0-e1f58482ac21\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zt46p"
Oct 30 19:28:54.066693 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:54.066608    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92500b73-8bdb-4bd9-8770-2a1247803adb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02944da9-v956g\" (UID: \"92500b73-8bdb-4bd9-8770-2a1247803adb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02944da9-v956g"
Oct 30 19:28:54.066693 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:54.066661    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f15d5348-e5a7-44d7-9ba0-e1f58482ac21-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zt46p\" (UID: \"f15d5348-e5a7-44d7-9ba0-e1f58482ac21\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zt46p"
Oct 30 19:28:54.066863 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:54.066695    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqsln\" (UniqueName: \"kubernetes.io/projected/92500b73-8bdb-4bd9-8770-2a1247803adb-kube-api-access-cqsln\") pod \"maas-api-revoke-sub-02944da9-v956g\" (UID: \"92500b73-8bdb-4bd9-8770-2a1247803adb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02944da9-v956g"
Oct 30 19:28:54.066863 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:54.066723    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f15d5348-e5a7-44d7-9ba0-e1f58482ac21-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zt46p\" (UID: \"f15d5348-e5a7-44d7-9ba0-e1f58482ac21\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zt46p"
Oct 30 19:28:54.066863 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:54.066796    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfjfl\" (UniqueName: \"kubernetes.io/projected/f15d5348-e5a7-44d7-9ba0-e1f58482ac21-kube-api-access-rfjfl\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zt46p\" (UID: \"f15d5348-e5a7-44d7-9ba0-e1f58482ac21\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zt46p"
Oct 30 19:28:54.066996 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:54.066901    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f15d5348-e5a7-44d7-9ba0-e1f58482ac21-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zt46p\" (UID: \"f15d5348-e5a7-44d7-9ba0-e1f58482ac21\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zt46p"
Oct 30 19:28:54.067049 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:54.067010    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92500b73-8bdb-4bd9-8770-2a1247803adb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02944da9-v956g\" (UID: \"92500b73-8bdb-4bd9-8770-2a1247803adb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02944da9-v956g"
Oct 30 19:28:54.068654 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:54.068626    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92500b73-8bdb-4bd9-8770-2a1247803adb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02944da9-v956g\" (UID: \"92500b73-8bdb-4bd9-8770-2a1247803adb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02944da9-v956g"
Oct 30 19:28:54.070282 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:54.070256    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92500b73-8bdb-4bd9-8770-2a1247803adb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02944da9-v956g\" (UID: \"92500b73-8bdb-4bd9-8770-2a1247803adb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02944da9-v956g"
Oct 30 19:28:54.081833 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:54.081519    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqsln\" (UniqueName: \"kubernetes.io/projected/92500b73-8bdb-4bd9-8770-2a1247803adb-kube-api-access-cqsln\") pod \"maas-api-revoke-sub-02944da9-v956g\" (UID: \"92500b73-8bdb-4bd9-8770-2a1247803adb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02944da9-v956g"
Oct 30 19:28:54.160530 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:54.160496    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02944da9-v956g"
Oct 30 19:28:54.168501 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:54.168470    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f15d5348-e5a7-44d7-9ba0-e1f58482ac21-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zt46p\" (UID: \"f15d5348-e5a7-44d7-9ba0-e1f58482ac21\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zt46p"
Oct 30 19:28:54.168648 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:54.168513    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f15d5348-e5a7-44d7-9ba0-e1f58482ac21-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zt46p\" (UID: \"f15d5348-e5a7-44d7-9ba0-e1f58482ac21\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zt46p"
Oct 30 19:28:54.168648 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:54.168544    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f15d5348-e5a7-44d7-9ba0-e1f58482ac21-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zt46p\" (UID: \"f15d5348-e5a7-44d7-9ba0-e1f58482ac21\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zt46p"
Oct 30 19:28:54.168648 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:54.168620    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f15d5348-e5a7-44d7-9ba0-e1f58482ac21-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zt46p\" (UID: \"f15d5348-e5a7-44d7-9ba0-e1f58482ac21\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zt46p"
Oct 30 19:28:54.168863 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:54.168650    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f15d5348-e5a7-44d7-9ba0-e1f58482ac21-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zt46p\" (UID: \"f15d5348-e5a7-44d7-9ba0-e1f58482ac21\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zt46p"
Oct 30 19:28:54.168863 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:54.168684    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfjfl\" (UniqueName: \"kubernetes.io/projected/f15d5348-e5a7-44d7-9ba0-e1f58482ac21-kube-api-access-rfjfl\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zt46p\" (UID: \"f15d5348-e5a7-44d7-9ba0-e1f58482ac21\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zt46p"
Oct 30 19:28:54.168863 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:54.168805    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f15d5348-e5a7-44d7-9ba0-e1f58482ac21-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zt46p\" (UID: \"f15d5348-e5a7-44d7-9ba0-e1f58482ac21\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zt46p"
Oct 30 19:28:54.169902 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:54.169879    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f15d5348-e5a7-44d7-9ba0-e1f58482ac21-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zt46p\" (UID: \"f15d5348-e5a7-44d7-9ba0-e1f58482ac21\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zt46p"
Oct 30 19:28:54.174412 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:54.174382    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f15d5348-e5a7-44d7-9ba0-e1f58482ac21-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zt46p\" (UID: \"f15d5348-e5a7-44d7-9ba0-e1f58482ac21\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zt46p"
Oct 30 19:28:54.176597 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:54.176273    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f15d5348-e5a7-44d7-9ba0-e1f58482ac21-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zt46p\" (UID: \"f15d5348-e5a7-44d7-9ba0-e1f58482ac21\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zt46p"
Oct 30 19:28:54.178443 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:54.178420    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f15d5348-e5a7-44d7-9ba0-e1f58482ac21-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zt46p\" (UID: \"f15d5348-e5a7-44d7-9ba0-e1f58482ac21\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zt46p"
Oct 30 19:28:54.178972 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:54.178927    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f15d5348-e5a7-44d7-9ba0-e1f58482ac21-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zt46p\" (UID: \"f15d5348-e5a7-44d7-9ba0-e1f58482ac21\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zt46p"
Oct 30 19:28:54.179429 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:54.179406    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfjfl\" (UniqueName: \"kubernetes.io/projected/f15d5348-e5a7-44d7-9ba0-e1f58482ac21-kube-api-access-rfjfl\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zt46p\" (UID: \"f15d5348-e5a7-44d7-9ba0-e1f58482ac21\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zt46p"
Oct 30 19:28:54.186530 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:54.186485    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f15d5348-e5a7-44d7-9ba0-e1f58482ac21-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zt46p\" (UID: \"f15d5348-e5a7-44d7-9ba0-e1f58482ac21\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zt46p"
Oct 30 19:28:54.230477 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:54.230132    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zt46p"
Oct 30 19:28:54.475693 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:54.475660    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02944da9-v956g"]
Oct 30 19:28:54.476086 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:28:54.475915    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92500b73_8bdb_4bd9_8770_2a1247803adb.slice/crio-43d07df9323fd05289df50457ad11609aac85cc5bbbd5093ec5731673eefc8ad WatchSource:0}: Error finding container 43d07df9323fd05289df50457ad11609aac85cc5bbbd5093ec5731673eefc8ad: Status 404 returned error can't find the container with id 43d07df9323fd05289df50457ad11609aac85cc5bbbd5093ec5731673eefc8ad
Oct 30 19:28:54.621060 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:28:54.619055    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf15d5348_e5a7_44d7_9ba0_e1f58482ac21.slice/crio-20051f4928b40a9a539358e6966d90cf16054b628cd76d127b28609096593b1a WatchSource:0}: Error finding container 20051f4928b40a9a539358e6966d90cf16054b628cd76d127b28609096593b1a: Status 404 returned error can't find the container with id 20051f4928b40a9a539358e6966d90cf16054b628cd76d127b28609096593b1a
Oct 30 19:28:54.621060 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:54.619756    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zt46p"]
Oct 30 19:28:54.622048 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:54.622029    5043 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:28:55.100220 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:28:55.100153    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:28:55.375829 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:55.375529    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02944da9-v956g" event={"ID":"92500b73-8bdb-4bd9-8770-2a1247803adb","Type":"ContainerStarted","Data":"fa58e3f8dcfdb51a943f3ff23caeb05df56f525abc025ed8e4bd1a88e2fc06c0"}
Oct 30 19:28:55.375829 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:55.375594    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02944da9-v956g" event={"ID":"92500b73-8bdb-4bd9-8770-2a1247803adb","Type":"ContainerStarted","Data":"43d07df9323fd05289df50457ad11609aac85cc5bbbd5093ec5731673eefc8ad"}
Oct 30 19:28:55.387762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:55.384472    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zt46p" event={"ID":"f15d5348-e5a7-44d7-9ba0-e1f58482ac21","Type":"ContainerStarted","Data":"20051f4928b40a9a539358e6966d90cf16054b628cd76d127b28609096593b1a"}
Oct 30 19:28:55.444979 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:55.444910    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02944da9-v956g" podStartSLOduration=2.444885434 podStartE2EDuration="2.444885434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:55.443970433 +0000 UTC m=+754.955030400" watchObservedRunningTime="2026-10-30 19:28:55.444885434 +0000 UTC m=+754.955945394"
Oct 30 19:28:56.465765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:56.459118    5043 generic.go:356] "Generic (PLEG): container finished" podID="92500b73-8bdb-4bd9-8770-2a1247803adb" containerID="fa58e3f8dcfdb51a943f3ff23caeb05df56f525abc025ed8e4bd1a88e2fc06c0" exitCode=0
Oct 30 19:28:56.465765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:56.459190    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02944da9-v956g" event={"ID":"92500b73-8bdb-4bd9-8770-2a1247803adb","Type":"ContainerDied","Data":"fa58e3f8dcfdb51a943f3ff23caeb05df56f525abc025ed8e4bd1a88e2fc06c0"}
Oct 30 19:28:56.796827 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:56.786875    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:28:56.872790 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:56.842507    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j"]
Oct 30 19:28:56.963675 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:56.961634    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j"
Oct 30 19:28:56.963675 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:56.962411    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j"]
Oct 30 19:28:56.993866 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:56.992420    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:57.026763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:56.994132    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:57.034763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:57.032567    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-workload-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j\" (UID: \"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j"
Oct 30 19:28:57.034763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:57.032608    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-istio-podinfo\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j\" (UID: \"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j"
Oct 30 19:28:57.034763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:57.032684    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-istio-token\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j\" (UID: \"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j"
Oct 30 19:28:57.034763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:57.032710    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-istiod-ca-cert\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j\" (UID: \"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j"
Oct 30 19:28:57.034763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:57.032830    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-istio-envoy\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j\" (UID: \"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j"
Oct 30 19:28:57.034763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:57.032888    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-credential-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j\" (UID: \"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j"
Oct 30 19:28:57.034763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:57.032913    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvglx\" (UniqueName: \"kubernetes.io/projected/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-kube-api-access-qvglx\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j\" (UID: \"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j"
Oct 30 19:28:57.034763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:57.032983    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-workload-certs\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j\" (UID: \"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j"
Oct 30 19:28:57.034763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:57.033013    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-istio-data\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j\" (UID: \"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j"
Oct 30 19:28:57.164247 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:57.158861    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:57.164247 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:57.160277    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:57.164247 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:57.161275    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-istio-envoy\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j\" (UID: \"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j"
Oct 30 19:28:57.164247 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:57.161347    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-credential-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j\" (UID: \"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j"
Oct 30 19:28:57.164247 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:57.161381    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvglx\" (UniqueName: \"kubernetes.io/projected/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-kube-api-access-qvglx\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j\" (UID: \"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j"
Oct 30 19:28:57.164247 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:57.161467    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-workload-certs\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j\" (UID: \"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j"
Oct 30 19:28:57.164247 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:57.161501    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-istio-data\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j\" (UID: \"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j"
Oct 30 19:28:57.164247 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:57.161553    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-workload-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j\" (UID: \"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j"
Oct 30 19:28:57.164247 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:57.161578    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-istio-podinfo\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j\" (UID: \"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j"
Oct 30 19:28:57.164247 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:57.161679    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-istio-token\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j\" (UID: \"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j"
Oct 30 19:28:57.164247 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:57.161706    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-istiod-ca-cert\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j\" (UID: \"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j"
Oct 30 19:28:57.164247 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:57.162230    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-istio-data\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j\" (UID: \"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j"
Oct 30 19:28:57.164247 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:57.162487    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-credential-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j\" (UID: \"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j"
Oct 30 19:28:57.164247 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:57.162606    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-istiod-ca-cert\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j\" (UID: \"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j"
Oct 30 19:28:57.164247 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:57.163026    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-workload-certs\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j\" (UID: \"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j"
Oct 30 19:28:57.165814 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:57.165772    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-workload-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j\" (UID: \"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j"
Oct 30 19:28:57.220705 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:57.208260    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-istio-envoy\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j\" (UID: \"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j"
Oct 30 19:28:57.262863 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:57.250310    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 30 19:28:57.265029 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:57.263546    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:28:57.275494 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:57.275412    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-istio-podinfo\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j\" (UID: \"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j"
Oct 30 19:28:57.327409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:57.327359    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:57.336618 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:57.335616    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvglx\" (UniqueName: \"kubernetes.io/projected/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-kube-api-access-qvglx\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j\" (UID: \"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j"
Oct 30 19:28:57.347912 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:57.346948    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:28:57.418369 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:57.418254    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-istio-token\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j\" (UID: \"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j"
Oct 30 19:28:57.613761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:57.610040    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": context deadline exceeded"
Oct 30 19:28:57.710529 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:57.710491    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j"
Oct 30 19:28:57.791332 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:57.791290    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:57.810895 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:57.810851    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:57.812440 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:57.812415    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:28:58.025828 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:58.011823    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:58.025828 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:58.012253    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:28:58.138765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:58.105140    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:28:58.179764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:58.163949    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:58.256767 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:58.256480    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 30 19:28:58.341968 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:58.341653    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:58.349379 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:58.345749    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:58.381760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:58.378895    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:58.507114 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:58.164389    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:28:58.507114 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:58.506966    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"
Oct 30 19:28:58.507114 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:58.507052    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:28:58.507114 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:58.507112    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:28:58.507460 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:58.507166    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:28:58.989138 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:58.989074    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:28:59.191358 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:59.182306    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:28:59.203783 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:59.203146    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:28:59.209805 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:59.209775    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"
Oct 30 19:28:59.252771 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:59.244246    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:28:59.263310 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:59.253085    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:28:59.270425 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:59.270150    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:28:59.271207 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:59.271179    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:28:59.342247 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:59.342172    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:28:59.344789 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:59.344714    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02944da9-v956g"
Oct 30 19:28:59.348662 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:59.348633    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c057c0d-wtvt4"]
Oct 30 19:28:59.349659 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:59.349626    5043 cpu_manager.go:414] "containerMap: removing container" podUID="92500b73-8bdb-4bd9-8770-2a1247803adb" containerName="revoke-keys"
Oct 30 19:28:59.349659 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:59.349654    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92500b73-8bdb-4bd9-8770-2a1247803adb" containerName="revoke-keys"
Oct 30 19:28:59.350694 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:59.350676    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="92500b73-8bdb-4bd9-8770-2a1247803adb" containerName="revoke-keys"
Oct 30 19:28:59.362924 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:59.362898    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c057c0d-wtvt4"
Oct 30 19:28:59.388764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:59.388673    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c057c0d-wtvt4"]
Oct 30 19:28:59.507811 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:59.507631    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92500b73-8bdb-4bd9-8770-2a1247803adb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92500b73-8bdb-4bd9-8770-2a1247803adb-maas-api-xks-ca\") pod \"92500b73-8bdb-4bd9-8770-2a1247803adb\" (UID: \"92500b73-8bdb-4bd9-8770-2a1247803adb\") "
Oct 30 19:28:59.508083 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:59.508062    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92500b73-8bdb-4bd9-8770-2a1247803adb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92500b73-8bdb-4bd9-8770-2a1247803adb-maas-api-openshift-ca\") pod \"92500b73-8bdb-4bd9-8770-2a1247803adb\" (UID: \"92500b73-8bdb-4bd9-8770-2a1247803adb\") "
Oct 30 19:28:59.508218 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:59.508204    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92500b73-8bdb-4bd9-8770-2a1247803adb-kube-api-access-cqsln\" (UniqueName: \"kubernetes.io/projected/92500b73-8bdb-4bd9-8770-2a1247803adb-kube-api-access-cqsln\") pod \"92500b73-8bdb-4bd9-8770-2a1247803adb\" (UID: \"92500b73-8bdb-4bd9-8770-2a1247803adb\") "
Oct 30 19:28:59.513038 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:59.513010    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6db403a-9d0f-48a8-ad28-7c5b8eb3172e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c057c0d-wtvt4\" (UID: \"e6db403a-9d0f-48a8-ad28-7c5b8eb3172e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c057c0d-wtvt4"
Oct 30 19:28:59.516373 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:59.513480    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92500b73-8bdb-4bd9-8770-2a1247803adb-maas-api-openshift-ca" pod "92500b73-8bdb-4bd9-8770-2a1247803adb" (UID: "92500b73-8bdb-4bd9-8770-2a1247803adb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:28:59.516373 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:59.516257    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92500b73-8bdb-4bd9-8770-2a1247803adb-maas-api-xks-ca" pod "92500b73-8bdb-4bd9-8770-2a1247803adb" (UID: "92500b73-8bdb-4bd9-8770-2a1247803adb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:28:59.518203 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:59.518167    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6db403a-9d0f-48a8-ad28-7c5b8eb3172e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c057c0d-wtvt4\" (UID: \"e6db403a-9d0f-48a8-ad28-7c5b8eb3172e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c057c0d-wtvt4"
Oct 30 19:28:59.518324 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:59.518307    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ngw9\" (UniqueName: \"kubernetes.io/projected/e6db403a-9d0f-48a8-ad28-7c5b8eb3172e-kube-api-access-6ngw9\") pod \"maas-api-revoke-sub-8c057c0d-wtvt4\" (UID: \"e6db403a-9d0f-48a8-ad28-7c5b8eb3172e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c057c0d-wtvt4"
Oct 30 19:28:59.518601 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:59.518473    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92500b73-8bdb-4bd9-8770-2a1247803adb-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:59.518601 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:59.518493    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92500b73-8bdb-4bd9-8770-2a1247803adb-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:59.526493 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:59.526436    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92500b73-8bdb-4bd9-8770-2a1247803adb-kube-api-access-cqsln" pod "92500b73-8bdb-4bd9-8770-2a1247803adb" (UID: "92500b73-8bdb-4bd9-8770-2a1247803adb"). InnerVolumeSpecName "kube-api-access-cqsln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:28:59.620685 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:59.620500    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ngw9\" (UniqueName: \"kubernetes.io/projected/e6db403a-9d0f-48a8-ad28-7c5b8eb3172e-kube-api-access-6ngw9\") pod \"maas-api-revoke-sub-8c057c0d-wtvt4\" (UID: \"e6db403a-9d0f-48a8-ad28-7c5b8eb3172e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c057c0d-wtvt4"
Oct 30 19:28:59.620685 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:59.620610    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6db403a-9d0f-48a8-ad28-7c5b8eb3172e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c057c0d-wtvt4\" (UID: \"e6db403a-9d0f-48a8-ad28-7c5b8eb3172e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c057c0d-wtvt4"
Oct 30 19:28:59.620952 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:59.620713    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6db403a-9d0f-48a8-ad28-7c5b8eb3172e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c057c0d-wtvt4\" (UID: \"e6db403a-9d0f-48a8-ad28-7c5b8eb3172e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c057c0d-wtvt4"
Oct 30 19:28:59.620952 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:59.620810    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqsln\" (UniqueName: \"kubernetes.io/projected/92500b73-8bdb-4bd9-8770-2a1247803adb-kube-api-access-cqsln\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:28:59.622201 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:59.622172    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6db403a-9d0f-48a8-ad28-7c5b8eb3172e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c057c0d-wtvt4\" (UID: \"e6db403a-9d0f-48a8-ad28-7c5b8eb3172e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c057c0d-wtvt4"
Oct 30 19:28:59.625697 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:59.625669    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6db403a-9d0f-48a8-ad28-7c5b8eb3172e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c057c0d-wtvt4\" (UID: \"e6db403a-9d0f-48a8-ad28-7c5b8eb3172e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c057c0d-wtvt4"
Oct 30 19:28:59.650585 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:59.650546    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j"]
Oct 30 19:28:59.677690 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:28:59.677638    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3185a2a_d994_41d4_a8d7_9b7e966b9f1c.slice/crio-99a241c37b6357bab02d273b53fa120ad8c1b5bb46292bda0db2b06e2879d92a WatchSource:0}: Error finding container 99a241c37b6357bab02d273b53fa120ad8c1b5bb46292bda0db2b06e2879d92a: Status 404 returned error can't find the container with id 99a241c37b6357bab02d273b53fa120ad8c1b5bb46292bda0db2b06e2879d92a
Oct 30 19:28:59.689701 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:59.680654    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:28:59.689701 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:59.683868    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ngw9\" (UniqueName: \"kubernetes.io/projected/e6db403a-9d0f-48a8-ad28-7c5b8eb3172e-kube-api-access-6ngw9\") pod \"maas-api-revoke-sub-8c057c0d-wtvt4\" (UID: \"e6db403a-9d0f-48a8-ad28-7c5b8eb3172e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c057c0d-wtvt4"
Oct 30 19:28:59.690614 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:59.690024    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:28:59.690614 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:59.690115    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:28:59.693975 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:28:59.692063    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c057c0d-wtvt4"
Oct 30 19:29:00.034709 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:00.034626    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c057c0d-wtvt4"]
Oct 30 19:29:00.218748 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:00.218704    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c057c0d-wtvt4" event={"ID":"e6db403a-9d0f-48a8-ad28-7c5b8eb3172e","Type":"ContainerStarted","Data":"2200519aac3b24fe76e4ad88c0231b628e420bdec0929d42806eca2f0f9286b0"}
Oct 30 19:29:00.222686 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:00.222644    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j" event={"ID":"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c","Type":"ContainerStarted","Data":"20e7692b745948d12f8044909a2acb7baae7326b2556c005133d80f20e87bf91"}
Oct 30 19:29:00.222855 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:00.222706    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j" event={"ID":"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c","Type":"ContainerStarted","Data":"99a241c37b6357bab02d273b53fa120ad8c1b5bb46292bda0db2b06e2879d92a"}
Oct 30 19:29:00.233859 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:00.233830    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02944da9-v956g" event={"ID":"92500b73-8bdb-4bd9-8770-2a1247803adb","Type":"ContainerDied","Data":"43d07df9323fd05289df50457ad11609aac85cc5bbbd5093ec5731673eefc8ad"}
Oct 30 19:29:00.234020 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:00.233882    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43d07df9323fd05289df50457ad11609aac85cc5bbbd5093ec5731673eefc8ad"
Oct 30 19:29:00.234020 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:00.233896    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02944da9-v956g"
Oct 30 19:29:00.255041 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:00.251755    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j" podStartSLOduration=4.251723865 podStartE2EDuration="4.251723865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:00.249378059 +0000 UTC m=+759.760438026" watchObservedRunningTime="2026-10-30 19:29:00.251723865 +0000 UTC m=+759.762783919"
Oct 30 19:29:00.713300 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:00.712853    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j"
Oct 30 19:29:00.713484 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:00.713397    5043 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 19:29:00.713484 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:00.713446    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j" podUID="c3185a2a-d994-41d4-a8d7-9b7e966b9f1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 19:29:01.245998 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:01.245962    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c057c0d-wtvt4" event={"ID":"e6db403a-9d0f-48a8-ad28-7c5b8eb3172e","Type":"ContainerStarted","Data":"6599f50ffe00618be78a309b8e72facb237131b2aae0c17a192fa97afebe66ec"}
Oct 30 19:29:01.295545 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:01.287995    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c057c0d-wtvt4" podStartSLOduration=2.287970381 podStartE2EDuration="2.287970381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:01.263159445 +0000 UTC m=+760.774219412" watchObservedRunningTime="2026-10-30 19:29:01.287970381 +0000 UTC m=+760.799030348"
Oct 30 19:29:01.712917 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:01.712816    5043 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 19:29:01.713098 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:01.712944    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j" podUID="c3185a2a-d994-41d4-a8d7-9b7e966b9f1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 19:29:02.054927 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:02.054831    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"
Oct 30 19:29:02.111232 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:02.111189    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"
Oct 30 19:29:02.315582 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:02.315321    5043 generic.go:356] "Generic (PLEG): container finished" podID="e6db403a-9d0f-48a8-ad28-7c5b8eb3172e" containerID="6599f50ffe00618be78a309b8e72facb237131b2aae0c17a192fa97afebe66ec" exitCode=0
Oct 30 19:29:02.315582 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:02.315477    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c057c0d-wtvt4" event={"ID":"e6db403a-9d0f-48a8-ad28-7c5b8eb3172e","Type":"ContainerDied","Data":"6599f50ffe00618be78a309b8e72facb237131b2aae0c17a192fa97afebe66ec"}
Oct 30 19:29:02.335566 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:02.335544    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"
Oct 30 19:29:02.721956 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:02.713328    5043 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 19:29:02.721956 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:02.713417    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j" podUID="c3185a2a-d994-41d4-a8d7-9b7e966b9f1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 19:29:03.715076 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:03.715022    5043 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 19:29:03.715556 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:03.715096    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j" podUID="c3185a2a-d994-41d4-a8d7-9b7e966b9f1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 19:29:03.723366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:03.723337    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c057c0d-wtvt4"
Oct 30 19:29:03.807799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:03.805842    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01637b35-5sckc"]
Oct 30 19:29:03.807799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:03.806796    5043 cpu_manager.go:414] "containerMap: removing container" podUID="e6db403a-9d0f-48a8-ad28-7c5b8eb3172e" containerName="revoke-keys"
Oct 30 19:29:03.807799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:03.806812    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6db403a-9d0f-48a8-ad28-7c5b8eb3172e" containerName="revoke-keys"
Oct 30 19:29:03.807799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:03.806937    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6db403a-9d0f-48a8-ad28-7c5b8eb3172e" containerName="revoke-keys"
Oct 30 19:29:03.814091 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:03.812876    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01637b35-5sckc"]
Oct 30 19:29:03.814091 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:03.812964    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01637b35-5sckc"
Oct 30 19:29:03.837131 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:03.837106    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6db403a-9d0f-48a8-ad28-7c5b8eb3172e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6db403a-9d0f-48a8-ad28-7c5b8eb3172e-maas-api-openshift-ca\") pod \"e6db403a-9d0f-48a8-ad28-7c5b8eb3172e\" (UID: \"e6db403a-9d0f-48a8-ad28-7c5b8eb3172e\") "
Oct 30 19:29:03.837280 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:03.837157    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6db403a-9d0f-48a8-ad28-7c5b8eb3172e-kube-api-access-6ngw9\" (UniqueName: \"kubernetes.io/projected/e6db403a-9d0f-48a8-ad28-7c5b8eb3172e-kube-api-access-6ngw9\") pod \"e6db403a-9d0f-48a8-ad28-7c5b8eb3172e\" (UID: \"e6db403a-9d0f-48a8-ad28-7c5b8eb3172e\") "
Oct 30 19:29:03.837280 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:03.837226    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6db403a-9d0f-48a8-ad28-7c5b8eb3172e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6db403a-9d0f-48a8-ad28-7c5b8eb3172e-maas-api-xks-ca\") pod \"e6db403a-9d0f-48a8-ad28-7c5b8eb3172e\" (UID: \"e6db403a-9d0f-48a8-ad28-7c5b8eb3172e\") "
Oct 30 19:29:03.840808 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:03.837517    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6db403a-9d0f-48a8-ad28-7c5b8eb3172e-maas-api-openshift-ca" pod "e6db403a-9d0f-48a8-ad28-7c5b8eb3172e" (UID: "e6db403a-9d0f-48a8-ad28-7c5b8eb3172e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:29:03.840808 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:03.837908    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6db403a-9d0f-48a8-ad28-7c5b8eb3172e-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:03.840808 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:03.840674    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6db403a-9d0f-48a8-ad28-7c5b8eb3172e-kube-api-access-6ngw9" pod "e6db403a-9d0f-48a8-ad28-7c5b8eb3172e" (UID: "e6db403a-9d0f-48a8-ad28-7c5b8eb3172e"). InnerVolumeSpecName "kube-api-access-6ngw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:29:03.840808 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:03.841246    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6db403a-9d0f-48a8-ad28-7c5b8eb3172e-maas-api-xks-ca" pod "e6db403a-9d0f-48a8-ad28-7c5b8eb3172e" (UID: "e6db403a-9d0f-48a8-ad28-7c5b8eb3172e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:29:03.938869 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:03.938789    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01637b35-5sckc\" (UID: \"0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01637b35-5sckc"
Oct 30 19:29:03.938869 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:03.938832    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01637b35-5sckc\" (UID: \"0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01637b35-5sckc"
Oct 30 19:29:03.939058 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:03.938899    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz6pk\" (UniqueName: \"kubernetes.io/projected/0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d-kube-api-access-rz6pk\") pod \"maas-api-revoke-sub-01637b35-5sckc\" (UID: \"0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01637b35-5sckc"
Oct 30 19:29:03.939200 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:03.939180    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6db403a-9d0f-48a8-ad28-7c5b8eb3172e-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:03.939236 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:03.939204    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ngw9\" (UniqueName: \"kubernetes.io/projected/e6db403a-9d0f-48a8-ad28-7c5b8eb3172e-kube-api-access-6ngw9\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:04.035887 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:04.035859    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:29:04.035976 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:04.035919    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:29:04.040609 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:04.040587    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01637b35-5sckc\" (UID: \"0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01637b35-5sckc"
Oct 30 19:29:04.040711 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:04.040619    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01637b35-5sckc\" (UID: \"0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01637b35-5sckc"
Oct 30 19:29:04.040711 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:04.040644    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz6pk\" (UniqueName: \"kubernetes.io/projected/0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d-kube-api-access-rz6pk\") pod \"maas-api-revoke-sub-01637b35-5sckc\" (UID: \"0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01637b35-5sckc"
Oct 30 19:29:04.041229 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:04.041207    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01637b35-5sckc\" (UID: \"0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01637b35-5sckc"
Oct 30 19:29:04.043073 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:04.043053    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01637b35-5sckc\" (UID: \"0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01637b35-5sckc"
Oct 30 19:29:04.048928 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:04.048907    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz6pk\" (UniqueName: \"kubernetes.io/projected/0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d-kube-api-access-rz6pk\") pod \"maas-api-revoke-sub-01637b35-5sckc\" (UID: \"0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01637b35-5sckc"
Oct 30 19:29:04.129833 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:04.129802    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01637b35-5sckc"
Oct 30 19:29:04.343318 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:04.343278    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c057c0d-wtvt4" event={"ID":"e6db403a-9d0f-48a8-ad28-7c5b8eb3172e","Type":"ContainerDied","Data":"2200519aac3b24fe76e4ad88c0231b628e420bdec0929d42806eca2f0f9286b0"}
Oct 30 19:29:04.343470 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:04.343332    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2200519aac3b24fe76e4ad88c0231b628e420bdec0929d42806eca2f0f9286b0"
Oct 30 19:29:04.343470 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:04.343387    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c057c0d-wtvt4"
Oct 30 19:29:04.348405 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:04.347320    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zt46p" event={"ID":"f15d5348-e5a7-44d7-9ba0-e1f58482ac21","Type":"ContainerStarted","Data":"db2f606853320d696c2e1b86dadaab6512c428a356f499b0f2d5a74fb226db4e"}
Oct 30 19:29:04.348405 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:04.348353    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zt46p"
Oct 30 19:29:04.405221 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:04.405147    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zt46p" podStartSLOduration=1.9926734320000001 podStartE2EDuration="11.405126681s" totalImagesPullingTime="9.412453249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:04.367083818 +0000 UTC m=+763.878143787" watchObservedRunningTime="2026-10-30 19:29:04.405126681 +0000 UTC m=+763.916186648"
Oct 30 19:29:04.509986 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:04.509960    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01637b35-5sckc"]
Oct 30 19:29:04.712762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:04.712259    5043 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 19:29:04.712762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:04.712319    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j" podUID="c3185a2a-d994-41d4-a8d7-9b7e966b9f1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 19:29:05.369760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:05.360709    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01637b35-5sckc" event={"ID":"0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d","Type":"ContainerStarted","Data":"c8cfefb833025748cd52190194964a2bbf0a0efb0f79bd5243f55381c95e4c72"}
Oct 30 19:29:05.369760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:05.360785    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01637b35-5sckc" event={"ID":"0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d","Type":"ContainerStarted","Data":"b14e6fbbbc95a02f73ea5fe0b01dfaac8500c252b301850d61c77717e368487c"}
Oct 30 19:29:05.372768 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:05.370692    5043 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-zt46p container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:29:05.372768 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:05.370769    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zt46p" podUID="f15d5348-e5a7-44d7-9ba0-e1f58482ac21" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:29:05.395755 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:05.395572    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01637b35-5sckc" podStartSLOduration=2.395553139 podStartE2EDuration="2.395553139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:05.393795101 +0000 UTC m=+764.904855067" watchObservedRunningTime="2026-10-30 19:29:05.395553139 +0000 UTC m=+764.906613105"
Oct 30 19:29:05.712579 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:05.712534    5043 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 19:29:05.712775 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:05.712600    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j" podUID="c3185a2a-d994-41d4-a8d7-9b7e966b9f1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 19:29:06.370430 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:06.370390    5043 generic.go:356] "Generic (PLEG): container finished" podID="0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d" containerID="c8cfefb833025748cd52190194964a2bbf0a0efb0f79bd5243f55381c95e4c72" exitCode=0
Oct 30 19:29:06.370975 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:06.370469    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01637b35-5sckc" event={"ID":"0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d","Type":"ContainerDied","Data":"c8cfefb833025748cd52190194964a2bbf0a0efb0f79bd5243f55381c95e4c72"}
Oct 30 19:29:06.371431 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:06.371409    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zt46p"
Oct 30 19:29:06.712568 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:06.712537    5043 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 19:29:06.712723 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:06.712596    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j" podUID="c3185a2a-d994-41d4-a8d7-9b7e966b9f1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 19:29:07.612849 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:07.612825    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01637b35-5sckc"
Oct 30 19:29:07.667279 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:07.667186    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d-maas-api-openshift-ca\") pod \"0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d\" (UID: \"0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d\") "
Oct 30 19:29:07.667279 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:07.667230    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d-maas-api-xks-ca\") pod \"0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d\" (UID: \"0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d\") "
Oct 30 19:29:07.667517 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:07.667379    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d-kube-api-access-rz6pk\" (UniqueName: \"kubernetes.io/projected/0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d-kube-api-access-rz6pk\") pod \"0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d\" (UID: \"0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d\") "
Oct 30 19:29:07.667989 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:07.667958    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d-maas-api-openshift-ca" pod "0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d" (UID: "0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:29:07.671027 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:07.671002    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d-kube-api-access-rz6pk" pod "0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d" (UID: "0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d"). InnerVolumeSpecName "kube-api-access-rz6pk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:29:07.671167 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:07.671067    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d-maas-api-xks-ca" pod "0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d" (UID: "0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:29:07.717526 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:07.717503    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j"
Oct 30 19:29:07.717812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:07.717790    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j"
Oct 30 19:29:07.718661 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:07.718644    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j"
Oct 30 19:29:07.769848 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:07.769811    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:07.770098 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:07.770078    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:07.770225 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:07.770210    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz6pk\" (UniqueName: \"kubernetes.io/projected/0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d-kube-api-access-rz6pk\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:08.149909 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:08.149864    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b9ff6ab-c5t78"]
Oct 30 19:29:08.150497 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:08.150482    5043 cpu_manager.go:414] "containerMap: removing container" podUID="0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d" containerName="revoke-keys"
Oct 30 19:29:08.150575 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:08.150500    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d" containerName="revoke-keys"
Oct 30 19:29:08.150640 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:08.150630    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d" containerName="revoke-keys"
Oct 30 19:29:08.155527 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:08.155499    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9ff6ab-c5t78"
Oct 30 19:29:08.166992 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:08.166965    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b9ff6ab-c5t78"]
Oct 30 19:29:08.176003 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:08.175969    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq9h7\" (UniqueName: \"kubernetes.io/projected/199728c5-1b99-4baa-b45b-e0470005d82f-kube-api-access-cq9h7\") pod \"maas-api-revoke-sub-4b9ff6ab-c5t78\" (UID: \"199728c5-1b99-4baa-b45b-e0470005d82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9ff6ab-c5t78"
Oct 30 19:29:08.176339 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:08.176319    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/199728c5-1b99-4baa-b45b-e0470005d82f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b9ff6ab-c5t78\" (UID: \"199728c5-1b99-4baa-b45b-e0470005d82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9ff6ab-c5t78"
Oct 30 19:29:08.177111 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:08.177088    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/199728c5-1b99-4baa-b45b-e0470005d82f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b9ff6ab-c5t78\" (UID: \"199728c5-1b99-4baa-b45b-e0470005d82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9ff6ab-c5t78"
Oct 30 19:29:08.278970 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:08.278938    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/199728c5-1b99-4baa-b45b-e0470005d82f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b9ff6ab-c5t78\" (UID: \"199728c5-1b99-4baa-b45b-e0470005d82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9ff6ab-c5t78"
Oct 30 19:29:08.279197 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:08.279012    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq9h7\" (UniqueName: \"kubernetes.io/projected/199728c5-1b99-4baa-b45b-e0470005d82f-kube-api-access-cq9h7\") pod \"maas-api-revoke-sub-4b9ff6ab-c5t78\" (UID: \"199728c5-1b99-4baa-b45b-e0470005d82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9ff6ab-c5t78"
Oct 30 19:29:08.279197 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:08.279086    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/199728c5-1b99-4baa-b45b-e0470005d82f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b9ff6ab-c5t78\" (UID: \"199728c5-1b99-4baa-b45b-e0470005d82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9ff6ab-c5t78"
Oct 30 19:29:08.279968 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:08.279944    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/199728c5-1b99-4baa-b45b-e0470005d82f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b9ff6ab-c5t78\" (UID: \"199728c5-1b99-4baa-b45b-e0470005d82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9ff6ab-c5t78"
Oct 30 19:29:08.282086 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:08.282056    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/199728c5-1b99-4baa-b45b-e0470005d82f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b9ff6ab-c5t78\" (UID: \"199728c5-1b99-4baa-b45b-e0470005d82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9ff6ab-c5t78"
Oct 30 19:29:08.289260 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:08.289235    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq9h7\" (UniqueName: \"kubernetes.io/projected/199728c5-1b99-4baa-b45b-e0470005d82f-kube-api-access-cq9h7\") pod \"maas-api-revoke-sub-4b9ff6ab-c5t78\" (UID: \"199728c5-1b99-4baa-b45b-e0470005d82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9ff6ab-c5t78"
Oct 30 19:29:08.395468 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:08.395439    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01637b35-5sckc"
Oct 30 19:29:08.395468 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:08.395449    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01637b35-5sckc" event={"ID":"0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d","Type":"ContainerDied","Data":"b14e6fbbbc95a02f73ea5fe0b01dfaac8500c252b301850d61c77717e368487c"}
Oct 30 19:29:08.395719 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:08.395511    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b14e6fbbbc95a02f73ea5fe0b01dfaac8500c252b301850d61c77717e368487c"
Oct 30 19:29:08.475866 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:08.475831    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9ff6ab-c5t78"
Oct 30 19:29:08.835852 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:08.835821    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b9ff6ab-c5t78"]
Oct 30 19:29:08.842940 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:29:08.842902    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod199728c5_1b99_4baa_b45b_e0470005d82f.slice/crio-82a76e882648ee44268815776f85acb58ce4be1e916c344531621801ba49dca7 WatchSource:0}: Error finding container 82a76e882648ee44268815776f85acb58ce4be1e916c344531621801ba49dca7: Status 404 returned error can't find the container with id 82a76e882648ee44268815776f85acb58ce4be1e916c344531621801ba49dca7
Oct 30 19:29:09.405652 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:09.405614    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9ff6ab-c5t78" event={"ID":"199728c5-1b99-4baa-b45b-e0470005d82f","Type":"ContainerStarted","Data":"0de2733df249e7c88fe7602be83e4e4ba4dd3bc076639286d4019cb610e731a0"}
Oct 30 19:29:09.405861 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:09.405684    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9ff6ab-c5t78" event={"ID":"199728c5-1b99-4baa-b45b-e0470005d82f","Type":"ContainerStarted","Data":"82a76e882648ee44268815776f85acb58ce4be1e916c344531621801ba49dca7"}
Oct 30 19:29:09.428817 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:09.428773    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9ff6ab-c5t78" podStartSLOduration=1.4287600839999999 podStartE2EDuration="1.428760084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:09.424536705 +0000 UTC m=+768.935596671" watchObservedRunningTime="2026-10-30 19:29:09.428760084 +0000 UTC m=+768.939820049"
Oct 30 19:29:10.419125 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:10.419081    5043 generic.go:356] "Generic (PLEG): container finished" podID="199728c5-1b99-4baa-b45b-e0470005d82f" containerID="0de2733df249e7c88fe7602be83e4e4ba4dd3bc076639286d4019cb610e731a0" exitCode=0
Oct 30 19:29:10.419970 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:10.419941    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9ff6ab-c5t78" event={"ID":"199728c5-1b99-4baa-b45b-e0470005d82f","Type":"ContainerDied","Data":"0de2733df249e7c88fe7602be83e4e4ba4dd3bc076639286d4019cb610e731a0"}
Oct 30 19:29:11.319861 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.319820    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6b64735-c5zv6"]
Oct 30 19:29:11.325314 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.325257    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b64735-c5zv6"
Oct 30 19:29:11.331391 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.331359    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6b64735-c5zv6"]
Oct 30 19:29:11.426962 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.426923    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cd8aaed-488b-4236-b372-05d2c3236af1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6b64735-c5zv6\" (UID: \"5cd8aaed-488b-4236-b372-05d2c3236af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b64735-c5zv6"
Oct 30 19:29:11.427394 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.426971    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cd8aaed-488b-4236-b372-05d2c3236af1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6b64735-c5zv6\" (UID: \"5cd8aaed-488b-4236-b372-05d2c3236af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b64735-c5zv6"
Oct 30 19:29:11.427394 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.427087    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2fsc\" (UniqueName: \"kubernetes.io/projected/5cd8aaed-488b-4236-b372-05d2c3236af1-kube-api-access-c2fsc\") pod \"maas-api-revoke-sub-d6b64735-c5zv6\" (UID: \"5cd8aaed-488b-4236-b372-05d2c3236af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b64735-c5zv6"
Oct 30 19:29:11.528886 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.528851    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cd8aaed-488b-4236-b372-05d2c3236af1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6b64735-c5zv6\" (UID: \"5cd8aaed-488b-4236-b372-05d2c3236af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b64735-c5zv6"
Oct 30 19:29:11.529071 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.528893    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cd8aaed-488b-4236-b372-05d2c3236af1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6b64735-c5zv6\" (UID: \"5cd8aaed-488b-4236-b372-05d2c3236af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b64735-c5zv6"
Oct 30 19:29:11.529071 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.528947    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2fsc\" (UniqueName: \"kubernetes.io/projected/5cd8aaed-488b-4236-b372-05d2c3236af1-kube-api-access-c2fsc\") pod \"maas-api-revoke-sub-d6b64735-c5zv6\" (UID: \"5cd8aaed-488b-4236-b372-05d2c3236af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b64735-c5zv6"
Oct 30 19:29:11.529880 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.529765    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cd8aaed-488b-4236-b372-05d2c3236af1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6b64735-c5zv6\" (UID: \"5cd8aaed-488b-4236-b372-05d2c3236af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b64735-c5zv6"
Oct 30 19:29:11.531905 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.531880    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cd8aaed-488b-4236-b372-05d2c3236af1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6b64735-c5zv6\" (UID: \"5cd8aaed-488b-4236-b372-05d2c3236af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b64735-c5zv6"
Oct 30 19:29:11.539041 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.538978    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2fsc\" (UniqueName: \"kubernetes.io/projected/5cd8aaed-488b-4236-b372-05d2c3236af1-kube-api-access-c2fsc\") pod \"maas-api-revoke-sub-d6b64735-c5zv6\" (UID: \"5cd8aaed-488b-4236-b372-05d2c3236af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b64735-c5zv6"
Oct 30 19:29:11.649966 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.649931    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b64735-c5zv6"
Oct 30 19:29:11.666628 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.666598    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9ff6ab-c5t78"
Oct 30 19:29:11.734443 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.732382    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/199728c5-1b99-4baa-b45b-e0470005d82f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/199728c5-1b99-4baa-b45b-e0470005d82f-maas-api-openshift-ca\") pod \"199728c5-1b99-4baa-b45b-e0470005d82f\" (UID: \"199728c5-1b99-4baa-b45b-e0470005d82f\") "
Oct 30 19:29:11.734443 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.732550    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/199728c5-1b99-4baa-b45b-e0470005d82f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/199728c5-1b99-4baa-b45b-e0470005d82f-maas-api-xks-ca\") pod \"199728c5-1b99-4baa-b45b-e0470005d82f\" (UID: \"199728c5-1b99-4baa-b45b-e0470005d82f\") "
Oct 30 19:29:11.734443 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.732612    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/199728c5-1b99-4baa-b45b-e0470005d82f-kube-api-access-cq9h7\" (UniqueName: \"kubernetes.io/projected/199728c5-1b99-4baa-b45b-e0470005d82f-kube-api-access-cq9h7\") pod \"199728c5-1b99-4baa-b45b-e0470005d82f\" (UID: \"199728c5-1b99-4baa-b45b-e0470005d82f\") "
Oct 30 19:29:11.738161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.738128    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/199728c5-1b99-4baa-b45b-e0470005d82f-maas-api-openshift-ca" pod "199728c5-1b99-4baa-b45b-e0470005d82f" (UID: "199728c5-1b99-4baa-b45b-e0470005d82f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:29:11.743904 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.743850    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/199728c5-1b99-4baa-b45b-e0470005d82f-kube-api-access-cq9h7" pod "199728c5-1b99-4baa-b45b-e0470005d82f" (UID: "199728c5-1b99-4baa-b45b-e0470005d82f"). InnerVolumeSpecName "kube-api-access-cq9h7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:29:11.768364 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.768321    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/199728c5-1b99-4baa-b45b-e0470005d82f-maas-api-xks-ca" pod "199728c5-1b99-4baa-b45b-e0470005d82f" (UID: "199728c5-1b99-4baa-b45b-e0470005d82f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:29:11.784730 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.784481    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf"]
Oct 30 19:29:11.786026 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.785274    5043 cpu_manager.go:414] "containerMap: removing container" podUID="199728c5-1b99-4baa-b45b-e0470005d82f" containerName="revoke-keys"
Oct 30 19:29:11.786026 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.785291    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="199728c5-1b99-4baa-b45b-e0470005d82f" containerName="revoke-keys"
Oct 30 19:29:11.786026 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.785415    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="199728c5-1b99-4baa-b45b-e0470005d82f" containerName="revoke-keys"
Oct 30 19:29:11.793122 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.793042    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf"
Oct 30 19:29:11.807106 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.806978    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf"]
Oct 30 19:29:11.835883 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.835452    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a62f36bd-5442-4679-8616-5b75487cf90a-workload-certs\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf\" (UID: \"a62f36bd-5442-4679-8616-5b75487cf90a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf"
Oct 30 19:29:11.835883 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.835495    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a62f36bd-5442-4679-8616-5b75487cf90a-istiod-ca-cert\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf\" (UID: \"a62f36bd-5442-4679-8616-5b75487cf90a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf"
Oct 30 19:29:11.835883 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.835526    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a62f36bd-5442-4679-8616-5b75487cf90a-credential-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf\" (UID: \"a62f36bd-5442-4679-8616-5b75487cf90a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf"
Oct 30 19:29:11.835883 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.835616    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a62f36bd-5442-4679-8616-5b75487cf90a-istio-data\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf\" (UID: \"a62f36bd-5442-4679-8616-5b75487cf90a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf"
Oct 30 19:29:11.835883 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.835637    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq6fh\" (UniqueName: \"kubernetes.io/projected/a62f36bd-5442-4679-8616-5b75487cf90a-kube-api-access-rq6fh\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf\" (UID: \"a62f36bd-5442-4679-8616-5b75487cf90a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf"
Oct 30 19:29:11.835883 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.835693    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a62f36bd-5442-4679-8616-5b75487cf90a-workload-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf\" (UID: \"a62f36bd-5442-4679-8616-5b75487cf90a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf"
Oct 30 19:29:11.835883 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.835727    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a62f36bd-5442-4679-8616-5b75487cf90a-istio-envoy\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf\" (UID: \"a62f36bd-5442-4679-8616-5b75487cf90a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf"
Oct 30 19:29:11.835883 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.835763    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a62f36bd-5442-4679-8616-5b75487cf90a-istio-podinfo\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf\" (UID: \"a62f36bd-5442-4679-8616-5b75487cf90a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf"
Oct 30 19:29:11.836342 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.835975    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a62f36bd-5442-4679-8616-5b75487cf90a-istio-token\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf\" (UID: \"a62f36bd-5442-4679-8616-5b75487cf90a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf"
Oct 30 19:29:11.836342 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.836098    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq9h7\" (UniqueName: \"kubernetes.io/projected/199728c5-1b99-4baa-b45b-e0470005d82f-kube-api-access-cq9h7\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:11.836342 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.836118    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/199728c5-1b99-4baa-b45b-e0470005d82f-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:11.836342 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.836132    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/199728c5-1b99-4baa-b45b-e0470005d82f-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:11.847831 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.847802    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc"]
Oct 30 19:29:11.852421 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.852334    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc"
Oct 30 19:29:11.864577 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.864548    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc"]
Oct 30 19:29:11.938014 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.937982    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a62f36bd-5442-4679-8616-5b75487cf90a-workload-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf\" (UID: \"a62f36bd-5442-4679-8616-5b75487cf90a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf"
Oct 30 19:29:11.938197 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.938035    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af56765-aede-4480-8b11-83646e0f5284-workload-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc\" (UID: \"7af56765-aede-4480-8b11-83646e0f5284\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc"
Oct 30 19:29:11.938197 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.938118    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a62f36bd-5442-4679-8616-5b75487cf90a-istio-envoy\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf\" (UID: \"a62f36bd-5442-4679-8616-5b75487cf90a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf"
Oct 30 19:29:11.938197 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.938143    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a62f36bd-5442-4679-8616-5b75487cf90a-istio-podinfo\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf\" (UID: \"a62f36bd-5442-4679-8616-5b75487cf90a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf"
Oct 30 19:29:11.938197 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.938188    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7af56765-aede-4480-8b11-83646e0f5284-istio-envoy\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc\" (UID: \"7af56765-aede-4480-8b11-83646e0f5284\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc"
Oct 30 19:29:11.938413 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.938248    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a62f36bd-5442-4679-8616-5b75487cf90a-istio-token\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf\" (UID: \"a62f36bd-5442-4679-8616-5b75487cf90a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf"
Oct 30 19:29:11.938413 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.938281    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7af56765-aede-4480-8b11-83646e0f5284-workload-certs\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc\" (UID: \"7af56765-aede-4480-8b11-83646e0f5284\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc"
Oct 30 19:29:11.938413 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.938308    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7af56765-aede-4480-8b11-83646e0f5284-istio-podinfo\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc\" (UID: \"7af56765-aede-4480-8b11-83646e0f5284\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc"
Oct 30 19:29:11.938564 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.938417    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a62f36bd-5442-4679-8616-5b75487cf90a-workload-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf\" (UID: \"a62f36bd-5442-4679-8616-5b75487cf90a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf"
Oct 30 19:29:11.938564 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.938418    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a62f36bd-5442-4679-8616-5b75487cf90a-workload-certs\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf\" (UID: \"a62f36bd-5442-4679-8616-5b75487cf90a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf"
Oct 30 19:29:11.938564 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.938540    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7af56765-aede-4480-8b11-83646e0f5284-istio-token\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc\" (UID: \"7af56765-aede-4480-8b11-83646e0f5284\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc"
Oct 30 19:29:11.938752 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.938578    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a62f36bd-5442-4679-8616-5b75487cf90a-istiod-ca-cert\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf\" (UID: \"a62f36bd-5442-4679-8616-5b75487cf90a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf"
Oct 30 19:29:11.938752 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.938625    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a62f36bd-5442-4679-8616-5b75487cf90a-credential-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf\" (UID: \"a62f36bd-5442-4679-8616-5b75487cf90a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf"
Oct 30 19:29:11.938752 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.938649    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7af56765-aede-4480-8b11-83646e0f5284-istiod-ca-cert\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc\" (UID: \"7af56765-aede-4480-8b11-83646e0f5284\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc"
Oct 30 19:29:11.938752 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.938677    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af56765-aede-4480-8b11-83646e0f5284-credential-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc\" (UID: \"7af56765-aede-4480-8b11-83646e0f5284\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc"
Oct 30 19:29:11.938752 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.938718    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a62f36bd-5442-4679-8616-5b75487cf90a-workload-certs\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf\" (UID: \"a62f36bd-5442-4679-8616-5b75487cf90a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf"
Oct 30 19:29:11.939885 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.938791    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7af56765-aede-4480-8b11-83646e0f5284-istio-data\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc\" (UID: \"7af56765-aede-4480-8b11-83646e0f5284\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc"
Oct 30 19:29:11.939885 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.938843    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a62f36bd-5442-4679-8616-5b75487cf90a-istio-data\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf\" (UID: \"a62f36bd-5442-4679-8616-5b75487cf90a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf"
Oct 30 19:29:11.939885 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.938877    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq6fh\" (UniqueName: \"kubernetes.io/projected/a62f36bd-5442-4679-8616-5b75487cf90a-kube-api-access-rq6fh\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf\" (UID: \"a62f36bd-5442-4679-8616-5b75487cf90a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf"
Oct 30 19:29:11.939885 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.938939    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfjlk\" (UniqueName: \"kubernetes.io/projected/7af56765-aede-4480-8b11-83646e0f5284-kube-api-access-hfjlk\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc\" (UID: \"7af56765-aede-4480-8b11-83646e0f5284\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc"
Oct 30 19:29:11.939885 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.939308    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a62f36bd-5442-4679-8616-5b75487cf90a-istiod-ca-cert\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf\" (UID: \"a62f36bd-5442-4679-8616-5b75487cf90a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf"
Oct 30 19:29:11.939885 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.939325    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a62f36bd-5442-4679-8616-5b75487cf90a-credential-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf\" (UID: \"a62f36bd-5442-4679-8616-5b75487cf90a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf"
Oct 30 19:29:11.939885 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.939464    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a62f36bd-5442-4679-8616-5b75487cf90a-istio-data\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf\" (UID: \"a62f36bd-5442-4679-8616-5b75487cf90a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf"
Oct 30 19:29:11.940650 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.940633    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a62f36bd-5442-4679-8616-5b75487cf90a-istio-envoy\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf\" (UID: \"a62f36bd-5442-4679-8616-5b75487cf90a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf"
Oct 30 19:29:11.941609 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.941593    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a62f36bd-5442-4679-8616-5b75487cf90a-istio-podinfo\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf\" (UID: \"a62f36bd-5442-4679-8616-5b75487cf90a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf"
Oct 30 19:29:11.948316 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.948295    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a62f36bd-5442-4679-8616-5b75487cf90a-istio-token\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf\" (UID: \"a62f36bd-5442-4679-8616-5b75487cf90a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf"
Oct 30 19:29:11.948660 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.948641    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq6fh\" (UniqueName: \"kubernetes.io/projected/a62f36bd-5442-4679-8616-5b75487cf90a-kube-api-access-rq6fh\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf\" (UID: \"a62f36bd-5442-4679-8616-5b75487cf90a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf"
Oct 30 19:29:11.960526 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:29:11.960500    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cd8aaed_488b_4236_b372_05d2c3236af1.slice/crio-8e3b85003f3bff53939a68d971a62a7fa72a956818f1f2ff038401c64c031547 WatchSource:0}: Error finding container 8e3b85003f3bff53939a68d971a62a7fa72a956818f1f2ff038401c64c031547: Status 404 returned error can't find the container with id 8e3b85003f3bff53939a68d971a62a7fa72a956818f1f2ff038401c64c031547
Oct 30 19:29:11.961204 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:11.961183    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6b64735-c5zv6"]
Oct 30 19:29:12.040330 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:12.040297    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7af56765-aede-4480-8b11-83646e0f5284-istio-envoy\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc\" (UID: \"7af56765-aede-4480-8b11-83646e0f5284\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc"
Oct 30 19:29:12.040528 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:12.040360    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7af56765-aede-4480-8b11-83646e0f5284-workload-certs\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc\" (UID: \"7af56765-aede-4480-8b11-83646e0f5284\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc"
Oct 30 19:29:12.040528 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:12.040377    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7af56765-aede-4480-8b11-83646e0f5284-istio-podinfo\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc\" (UID: \"7af56765-aede-4480-8b11-83646e0f5284\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc"
Oct 30 19:29:12.040528 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:12.040444    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7af56765-aede-4480-8b11-83646e0f5284-istio-token\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc\" (UID: \"7af56765-aede-4480-8b11-83646e0f5284\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc"
Oct 30 19:29:12.041344 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:12.040584    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7af56765-aede-4480-8b11-83646e0f5284-istiod-ca-cert\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc\" (UID: \"7af56765-aede-4480-8b11-83646e0f5284\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc"
Oct 30 19:29:12.041344 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:12.040627    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af56765-aede-4480-8b11-83646e0f5284-credential-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc\" (UID: \"7af56765-aede-4480-8b11-83646e0f5284\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc"
Oct 30 19:29:12.041344 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:12.040709    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7af56765-aede-4480-8b11-83646e0f5284-istio-data\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc\" (UID: \"7af56765-aede-4480-8b11-83646e0f5284\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc"
Oct 30 19:29:12.041344 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:12.040788    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7af56765-aede-4480-8b11-83646e0f5284-workload-certs\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc\" (UID: \"7af56765-aede-4480-8b11-83646e0f5284\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc"
Oct 30 19:29:12.041344 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:12.040810    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfjlk\" (UniqueName: \"kubernetes.io/projected/7af56765-aede-4480-8b11-83646e0f5284-kube-api-access-hfjlk\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc\" (UID: \"7af56765-aede-4480-8b11-83646e0f5284\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc"
Oct 30 19:29:12.041344 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:12.040885    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af56765-aede-4480-8b11-83646e0f5284-workload-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc\" (UID: \"7af56765-aede-4480-8b11-83646e0f5284\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc"
Oct 30 19:29:12.041344 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:12.041021    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af56765-aede-4480-8b11-83646e0f5284-credential-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc\" (UID: \"7af56765-aede-4480-8b11-83646e0f5284\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc"
Oct 30 19:29:12.041344 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:12.041071    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7af56765-aede-4480-8b11-83646e0f5284-istio-data\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc\" (UID: \"7af56765-aede-4480-8b11-83646e0f5284\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc"
Oct 30 19:29:12.041344 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:12.041144    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7af56765-aede-4480-8b11-83646e0f5284-istiod-ca-cert\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc\" (UID: \"7af56765-aede-4480-8b11-83646e0f5284\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc"
Oct 30 19:29:12.041344 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:12.041204    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af56765-aede-4480-8b11-83646e0f5284-workload-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc\" (UID: \"7af56765-aede-4480-8b11-83646e0f5284\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc"
Oct 30 19:29:12.042813 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:12.042789    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7af56765-aede-4480-8b11-83646e0f5284-istio-envoy\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc\" (UID: \"7af56765-aede-4480-8b11-83646e0f5284\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc"
Oct 30 19:29:12.043084 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:12.043069    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7af56765-aede-4480-8b11-83646e0f5284-istio-podinfo\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc\" (UID: \"7af56765-aede-4480-8b11-83646e0f5284\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc"
Oct 30 19:29:12.051619 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:12.051584    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7af56765-aede-4480-8b11-83646e0f5284-istio-token\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc\" (UID: \"7af56765-aede-4480-8b11-83646e0f5284\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc"
Oct 30 19:29:12.053424 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:12.053403    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfjlk\" (UniqueName: \"kubernetes.io/projected/7af56765-aede-4480-8b11-83646e0f5284-kube-api-access-hfjlk\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc\" (UID: \"7af56765-aede-4480-8b11-83646e0f5284\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc"
Oct 30 19:29:12.113176 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:12.113089    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf"
Oct 30 19:29:12.172755 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:12.172708    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc"
Oct 30 19:29:12.438365 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:12.438276    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b64735-c5zv6" event={"ID":"5cd8aaed-488b-4236-b372-05d2c3236af1","Type":"ContainerStarted","Data":"8e3b85003f3bff53939a68d971a62a7fa72a956818f1f2ff038401c64c031547"}
Oct 30 19:29:12.439868 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:12.439846    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9ff6ab-c5t78"
Oct 30 19:29:12.439954 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:12.439875    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b9ff6ab-c5t78" event={"ID":"199728c5-1b99-4baa-b45b-e0470005d82f","Type":"ContainerDied","Data":"82a76e882648ee44268815776f85acb58ce4be1e916c344531621801ba49dca7"}
Oct 30 19:29:12.439954 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:12.439916    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82a76e882648ee44268815776f85acb58ce4be1e916c344531621801ba49dca7"
Oct 30 19:29:12.451189 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:12.451165    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf"]
Oct 30 19:29:12.451294 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:29:12.451277    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda62f36bd_5442_4679_8616_5b75487cf90a.slice/crio-7d7878d9134ee3ff18cabff8749a63ef222b76f3d10d6fce92bf97798baac2c0 WatchSource:0}: Error finding container 7d7878d9134ee3ff18cabff8749a63ef222b76f3d10d6fce92bf97798baac2c0: Status 404 returned error can't find the container with id 7d7878d9134ee3ff18cabff8749a63ef222b76f3d10d6fce92bf97798baac2c0
Oct 30 19:29:12.453823 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:12.453798    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:29:12.453892 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:12.453858    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:29:12.453892 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:12.453884    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:29:12.511271 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:12.511241    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc"]
Oct 30 19:29:12.511467 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:29:12.511446    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7af56765_aede_4480_8b11_83646e0f5284.slice/crio-e6350aa154576a6431d096e157c1e237c1c0b6053eef9b1e57a280006196f37f WatchSource:0}: Error finding container e6350aa154576a6431d096e157c1e237c1c0b6053eef9b1e57a280006196f37f: Status 404 returned error can't find the container with id e6350aa154576a6431d096e157c1e237c1c0b6053eef9b1e57a280006196f37f
Oct 30 19:29:12.514199 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:12.514168    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:29:12.514295 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:12.514229    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:29:12.514295 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:12.514258    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:29:13.449877 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:13.449772    5043 generic.go:356] "Generic (PLEG): container finished" podID="5cd8aaed-488b-4236-b372-05d2c3236af1" containerID="d5a29266789cbf17a2ef17f9bbda4f67a58d1503efb480565719e49c151a5de0" exitCode=0
Oct 30 19:29:13.450328 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:13.449984    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b64735-c5zv6" event={"ID":"5cd8aaed-488b-4236-b372-05d2c3236af1","Type":"ContainerDied","Data":"d5a29266789cbf17a2ef17f9bbda4f67a58d1503efb480565719e49c151a5de0"}
Oct 30 19:29:13.451925 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:13.451898    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc" event={"ID":"7af56765-aede-4480-8b11-83646e0f5284","Type":"ContainerStarted","Data":"ccd2f4229ca1ee1682bef8db4f483e81232ed682ae0fa15d1f6b8c43ea141508"}
Oct 30 19:29:13.452075 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:13.451948    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc" event={"ID":"7af56765-aede-4480-8b11-83646e0f5284","Type":"ContainerStarted","Data":"e6350aa154576a6431d096e157c1e237c1c0b6053eef9b1e57a280006196f37f"}
Oct 30 19:29:13.453680 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:13.453656    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf" event={"ID":"a62f36bd-5442-4679-8616-5b75487cf90a","Type":"ContainerStarted","Data":"86e91c900aa0277e244ae2e463d59315e17fe21ca62e63c4573764dcbec73794"}
Oct 30 19:29:13.453816 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:13.453696    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf" event={"ID":"a62f36bd-5442-4679-8616-5b75487cf90a","Type":"ContainerStarted","Data":"7d7878d9134ee3ff18cabff8749a63ef222b76f3d10d6fce92bf97798baac2c0"}
Oct 30 19:29:13.541036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:13.540978    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc" podStartSLOduration=2.540957075 podStartE2EDuration="2.540957075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:13.523568635 +0000 UTC m=+773.034628602" watchObservedRunningTime="2026-10-30 19:29:13.540957075 +0000 UTC m=+773.052017044"
Oct 30 19:29:14.114592 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:14.113985    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf"
Oct 30 19:29:14.116532 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:14.116497    5043 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:29:14.116681 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:14.116559    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf" podUID="a62f36bd-5442-4679-8616-5b75487cf90a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 19:29:14.181761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:14.180722    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc"
Oct 30 19:29:14.181761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:14.180880    5043 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:29:14.181761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:14.180926    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc" podUID="7af56765-aede-4480-8b11-83646e0f5284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 19:29:14.946902 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:14.946875    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b64735-c5zv6"
Oct 30 19:29:14.976770 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:14.974131    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf" podStartSLOduration=3.974111177 podStartE2EDuration="3.974111177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:13.560989379 +0000 UTC m=+773.072049363" watchObservedRunningTime="2026-10-30 19:29:14.974111177 +0000 UTC m=+774.485171142"
Oct 30 19:29:15.034894 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:15.034863    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cd8aaed-488b-4236-b372-05d2c3236af1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cd8aaed-488b-4236-b372-05d2c3236af1-maas-api-openshift-ca\") pod \"5cd8aaed-488b-4236-b372-05d2c3236af1\" (UID: \"5cd8aaed-488b-4236-b372-05d2c3236af1\") "
Oct 30 19:29:15.035250 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:15.035232    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cd8aaed-488b-4236-b372-05d2c3236af1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cd8aaed-488b-4236-b372-05d2c3236af1-maas-api-xks-ca\") pod \"5cd8aaed-488b-4236-b372-05d2c3236af1\" (UID: \"5cd8aaed-488b-4236-b372-05d2c3236af1\") "
Oct 30 19:29:15.035361 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:15.035348    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cd8aaed-488b-4236-b372-05d2c3236af1-kube-api-access-c2fsc\" (UniqueName: \"kubernetes.io/projected/5cd8aaed-488b-4236-b372-05d2c3236af1-kube-api-access-c2fsc\") pod \"5cd8aaed-488b-4236-b372-05d2c3236af1\" (UID: \"5cd8aaed-488b-4236-b372-05d2c3236af1\") "
Oct 30 19:29:15.039679 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:15.038093    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cd8aaed-488b-4236-b372-05d2c3236af1-maas-api-openshift-ca" pod "5cd8aaed-488b-4236-b372-05d2c3236af1" (UID: "5cd8aaed-488b-4236-b372-05d2c3236af1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:29:15.040198 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:15.040178    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cd8aaed-488b-4236-b372-05d2c3236af1-kube-api-access-c2fsc" pod "5cd8aaed-488b-4236-b372-05d2c3236af1" (UID: "5cd8aaed-488b-4236-b372-05d2c3236af1"). InnerVolumeSpecName "kube-api-access-c2fsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:29:15.042836 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:15.042317    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cd8aaed-488b-4236-b372-05d2c3236af1-maas-api-xks-ca" pod "5cd8aaed-488b-4236-b372-05d2c3236af1" (UID: "5cd8aaed-488b-4236-b372-05d2c3236af1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:29:15.122174 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:15.122139    5043 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:29:15.122378 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:15.122202    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf" podUID="a62f36bd-5442-4679-8616-5b75487cf90a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 19:29:15.137573 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:15.137544    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cd8aaed-488b-4236-b372-05d2c3236af1-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:15.137573 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:15.137574    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cd8aaed-488b-4236-b372-05d2c3236af1-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:15.137770 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:15.137585    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2fsc\" (UniqueName: \"kubernetes.io/projected/5cd8aaed-488b-4236-b372-05d2c3236af1-kube-api-access-c2fsc\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:15.176673 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:15.176162    5043 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:29:15.176673 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:15.176227    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc" podUID="7af56765-aede-4480-8b11-83646e0f5284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 19:29:15.549870 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:15.549835    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b64735-c5zv6" event={"ID":"5cd8aaed-488b-4236-b372-05d2c3236af1","Type":"ContainerDied","Data":"8e3b85003f3bff53939a68d971a62a7fa72a956818f1f2ff038401c64c031547"}
Oct 30 19:29:15.550154 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:15.550134    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e3b85003f3bff53939a68d971a62a7fa72a956818f1f2ff038401c64c031547"
Oct 30 19:29:15.550289 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:15.549844    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b64735-c5zv6"
Oct 30 19:29:16.117762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:16.117175    5043 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:29:16.117762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:16.117246    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf" podUID="a62f36bd-5442-4679-8616-5b75487cf90a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 19:29:16.182764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:16.176103    5043 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:29:16.182764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:16.176172    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc" podUID="7af56765-aede-4480-8b11-83646e0f5284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 19:29:17.118762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:17.117889    5043 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:29:17.118762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:17.117969    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf" podUID="a62f36bd-5442-4679-8616-5b75487cf90a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 19:29:17.188761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:17.188250    5043 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:29:17.188761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:17.188341    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc" podUID="7af56765-aede-4480-8b11-83646e0f5284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 19:29:18.118760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:18.115472    5043 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:29:18.118760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:18.115628    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf" podUID="a62f36bd-5442-4679-8616-5b75487cf90a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 19:29:18.178764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:18.177484    5043 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:29:18.178764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:18.177538    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc" podUID="7af56765-aede-4480-8b11-83646e0f5284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 19:29:19.114255 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:19.114210    5043 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:29:19.114460 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:19.114270    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf" podUID="a62f36bd-5442-4679-8616-5b75487cf90a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 19:29:19.173499 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:19.173449    5043 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:29:19.173996 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:19.173537    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc" podUID="7af56765-aede-4480-8b11-83646e0f5284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 19:29:20.296838 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:20.296802    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf"
Oct 30 19:29:20.297609 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:20.297578    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf"
Oct 30 19:29:20.298619 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:20.298594    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf"
Oct 30 19:29:20.441810 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:20.441774    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc"
Oct 30 19:29:20.442695 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:20.442669    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc"
Oct 30 19:29:20.443451 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:20.443430    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc"
Oct 30 19:29:21.244365 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:21.244328    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66d1bdfd-n8fks"]
Oct 30 19:29:21.245308 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:21.245289    5043 cpu_manager.go:414] "containerMap: removing container" podUID="5cd8aaed-488b-4236-b372-05d2c3236af1" containerName="revoke-keys"
Oct 30 19:29:21.245385 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:21.245311    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cd8aaed-488b-4236-b372-05d2c3236af1" containerName="revoke-keys"
Oct 30 19:29:21.245465 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:21.245454    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cd8aaed-488b-4236-b372-05d2c3236af1" containerName="revoke-keys"
Oct 30 19:29:21.251010 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:21.250983    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d1bdfd-n8fks"
Oct 30 19:29:21.255026 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:21.254931    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:29:21.257469 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:21.255618    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:29:21.257469 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:21.256155    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:29:21.257469 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:21.256216    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:29:21.257714 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:21.257691    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66d1bdfd-n8fks"]
Oct 30 19:29:21.345488 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:21.345459    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01a3afe0-c101-49e1-ae81-66fff2d8bedf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66d1bdfd-n8fks\" (UID: \"01a3afe0-c101-49e1-ae81-66fff2d8bedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d1bdfd-n8fks"
Oct 30 19:29:21.345890 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:21.345581    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmx98\" (UniqueName: \"kubernetes.io/projected/01a3afe0-c101-49e1-ae81-66fff2d8bedf-kube-api-access-qmx98\") pod \"maas-api-revoke-sub-66d1bdfd-n8fks\" (UID: \"01a3afe0-c101-49e1-ae81-66fff2d8bedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d1bdfd-n8fks"
Oct 30 19:29:21.345890 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:21.345639    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01a3afe0-c101-49e1-ae81-66fff2d8bedf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66d1bdfd-n8fks\" (UID: \"01a3afe0-c101-49e1-ae81-66fff2d8bedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d1bdfd-n8fks"
Oct 30 19:29:21.447051 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:21.447012    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01a3afe0-c101-49e1-ae81-66fff2d8bedf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66d1bdfd-n8fks\" (UID: \"01a3afe0-c101-49e1-ae81-66fff2d8bedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d1bdfd-n8fks"
Oct 30 19:29:21.447200 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:21.447092    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmx98\" (UniqueName: \"kubernetes.io/projected/01a3afe0-c101-49e1-ae81-66fff2d8bedf-kube-api-access-qmx98\") pod \"maas-api-revoke-sub-66d1bdfd-n8fks\" (UID: \"01a3afe0-c101-49e1-ae81-66fff2d8bedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d1bdfd-n8fks"
Oct 30 19:29:21.447200 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:21.447131    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01a3afe0-c101-49e1-ae81-66fff2d8bedf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66d1bdfd-n8fks\" (UID: \"01a3afe0-c101-49e1-ae81-66fff2d8bedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d1bdfd-n8fks"
Oct 30 19:29:21.447797 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:21.447776    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01a3afe0-c101-49e1-ae81-66fff2d8bedf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66d1bdfd-n8fks\" (UID: \"01a3afe0-c101-49e1-ae81-66fff2d8bedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d1bdfd-n8fks"
Oct 30 19:29:21.449630 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:21.449606    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01a3afe0-c101-49e1-ae81-66fff2d8bedf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66d1bdfd-n8fks\" (UID: \"01a3afe0-c101-49e1-ae81-66fff2d8bedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d1bdfd-n8fks"
Oct 30 19:29:21.455575 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:21.455553    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmx98\" (UniqueName: \"kubernetes.io/projected/01a3afe0-c101-49e1-ae81-66fff2d8bedf-kube-api-access-qmx98\") pod \"maas-api-revoke-sub-66d1bdfd-n8fks\" (UID: \"01a3afe0-c101-49e1-ae81-66fff2d8bedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d1bdfd-n8fks"
Oct 30 19:29:21.577160 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:21.577067    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d1bdfd-n8fks"
Oct 30 19:29:21.865100 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:29:21.865069    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01a3afe0_c101_49e1_ae81_66fff2d8bedf.slice/crio-5f83b014429623180f19cc29126c73e104cdce0f625549f7732e48fccfa2ae35 WatchSource:0}: Error finding container 5f83b014429623180f19cc29126c73e104cdce0f625549f7732e48fccfa2ae35: Status 404 returned error can't find the container with id 5f83b014429623180f19cc29126c73e104cdce0f625549f7732e48fccfa2ae35
Oct 30 19:29:21.866242 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:21.866212    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66d1bdfd-n8fks"]
Oct 30 19:29:22.681092 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:22.681004    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d1bdfd-n8fks" event={"ID":"01a3afe0-c101-49e1-ae81-66fff2d8bedf","Type":"ContainerStarted","Data":"665924c22ea9b263bf4f17c595d90ffae391803420e63f3e6064fdabed42bd47"}
Oct 30 19:29:22.681092 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:22.681061    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d1bdfd-n8fks" event={"ID":"01a3afe0-c101-49e1-ae81-66fff2d8bedf","Type":"ContainerStarted","Data":"5f83b014429623180f19cc29126c73e104cdce0f625549f7732e48fccfa2ae35"}
Oct 30 19:29:22.702893 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:22.702841    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d1bdfd-n8fks" podStartSLOduration=1.7028264480000002 podStartE2EDuration="1.702826448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:22.699683741 +0000 UTC m=+782.210743707" watchObservedRunningTime="2026-10-30 19:29:22.702826448 +0000 UTC m=+782.213886437"
Oct 30 19:29:23.008583 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:23.008546    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ee18dc2-cmntm"]
Oct 30 19:29:23.012906 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:23.012882    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee18dc2-cmntm"
Oct 30 19:29:23.021546 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:23.021516    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ee18dc2-cmntm"]
Oct 30 19:29:23.083203 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:23.083162    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9c119ec-7866-483b-a775-e3438b62b386-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ee18dc2-cmntm\" (UID: \"b9c119ec-7866-483b-a775-e3438b62b386\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee18dc2-cmntm"
Oct 30 19:29:23.083203 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:23.083203    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9c119ec-7866-483b-a775-e3438b62b386-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ee18dc2-cmntm\" (UID: \"b9c119ec-7866-483b-a775-e3438b62b386\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee18dc2-cmntm"
Oct 30 19:29:23.083455 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:23.083407    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxzxb\" (UniqueName: \"kubernetes.io/projected/b9c119ec-7866-483b-a775-e3438b62b386-kube-api-access-sxzxb\") pod \"maas-api-revoke-sub-1ee18dc2-cmntm\" (UID: \"b9c119ec-7866-483b-a775-e3438b62b386\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee18dc2-cmntm"
Oct 30 19:29:23.185346 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:23.185312    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9c119ec-7866-483b-a775-e3438b62b386-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ee18dc2-cmntm\" (UID: \"b9c119ec-7866-483b-a775-e3438b62b386\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee18dc2-cmntm"
Oct 30 19:29:23.185346 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:23.185344    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9c119ec-7866-483b-a775-e3438b62b386-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ee18dc2-cmntm\" (UID: \"b9c119ec-7866-483b-a775-e3438b62b386\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee18dc2-cmntm"
Oct 30 19:29:23.185618 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:23.185424    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxzxb\" (UniqueName: \"kubernetes.io/projected/b9c119ec-7866-483b-a775-e3438b62b386-kube-api-access-sxzxb\") pod \"maas-api-revoke-sub-1ee18dc2-cmntm\" (UID: \"b9c119ec-7866-483b-a775-e3438b62b386\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee18dc2-cmntm"
Oct 30 19:29:23.186184 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:23.186155    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9c119ec-7866-483b-a775-e3438b62b386-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ee18dc2-cmntm\" (UID: \"b9c119ec-7866-483b-a775-e3438b62b386\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee18dc2-cmntm"
Oct 30 19:29:23.187963 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:23.187939    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9c119ec-7866-483b-a775-e3438b62b386-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ee18dc2-cmntm\" (UID: \"b9c119ec-7866-483b-a775-e3438b62b386\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee18dc2-cmntm"
Oct 30 19:29:23.194037 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:23.194006    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxzxb\" (UniqueName: \"kubernetes.io/projected/b9c119ec-7866-483b-a775-e3438b62b386-kube-api-access-sxzxb\") pod \"maas-api-revoke-sub-1ee18dc2-cmntm\" (UID: \"b9c119ec-7866-483b-a775-e3438b62b386\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee18dc2-cmntm"
Oct 30 19:29:23.331489 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:23.331410    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee18dc2-cmntm"
Oct 30 19:29:23.618456 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:29:23.618418    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9c119ec_7866_483b_a775_e3438b62b386.slice/crio-8567521d5b0c346c36617af1e161d42d48b6279e12cae6f7cf0b1e72647899e3 WatchSource:0}: Error finding container 8567521d5b0c346c36617af1e161d42d48b6279e12cae6f7cf0b1e72647899e3: Status 404 returned error can't find the container with id 8567521d5b0c346c36617af1e161d42d48b6279e12cae6f7cf0b1e72647899e3
Oct 30 19:29:23.619249 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:23.619219    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ee18dc2-cmntm"]
Oct 30 19:29:23.719148 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:23.719106    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee18dc2-cmntm" event={"ID":"b9c119ec-7866-483b-a775-e3438b62b386","Type":"ContainerStarted","Data":"8567521d5b0c346c36617af1e161d42d48b6279e12cae6f7cf0b1e72647899e3"}
Oct 30 19:29:23.721482 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:23.721456    5043 generic.go:356] "Generic (PLEG): container finished" podID="01a3afe0-c101-49e1-ae81-66fff2d8bedf" containerID="665924c22ea9b263bf4f17c595d90ffae391803420e63f3e6064fdabed42bd47" exitCode=0
Oct 30 19:29:23.721599 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:23.721489    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d1bdfd-n8fks" event={"ID":"01a3afe0-c101-49e1-ae81-66fff2d8bedf","Type":"ContainerDied","Data":"665924c22ea9b263bf4f17c595d90ffae391803420e63f3e6064fdabed42bd47"}
Oct 30 19:29:24.736987 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:24.736948    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee18dc2-cmntm" event={"ID":"b9c119ec-7866-483b-a775-e3438b62b386","Type":"ContainerStarted","Data":"5d1c5fd409794239876528ad39b0b06ef0f78b809eb7c91d1d11132305727474"}
Oct 30 19:29:25.264620 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:29:25.264562    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:29:25.380837 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:25.380812    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d1bdfd-n8fks"
Oct 30 19:29:25.418683 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:25.418615    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee18dc2-cmntm" podStartSLOduration=3.418590012 podStartE2EDuration="3.418590012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:24.785149976 +0000 UTC m=+784.296209942" watchObservedRunningTime="2026-10-30 19:29:25.418590012 +0000 UTC m=+784.929649985"
Oct 30 19:29:25.471026 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:25.470986    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01a3afe0-c101-49e1-ae81-66fff2d8bedf-kube-api-access-qmx98\" (UniqueName: \"kubernetes.io/projected/01a3afe0-c101-49e1-ae81-66fff2d8bedf-kube-api-access-qmx98\") pod \"01a3afe0-c101-49e1-ae81-66fff2d8bedf\" (UID: \"01a3afe0-c101-49e1-ae81-66fff2d8bedf\") "
Oct 30 19:29:25.471206 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:25.471045    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01a3afe0-c101-49e1-ae81-66fff2d8bedf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01a3afe0-c101-49e1-ae81-66fff2d8bedf-maas-api-xks-ca\") pod \"01a3afe0-c101-49e1-ae81-66fff2d8bedf\" (UID: \"01a3afe0-c101-49e1-ae81-66fff2d8bedf\") "
Oct 30 19:29:25.471206 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:25.471096    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01a3afe0-c101-49e1-ae81-66fff2d8bedf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01a3afe0-c101-49e1-ae81-66fff2d8bedf-maas-api-openshift-ca\") pod \"01a3afe0-c101-49e1-ae81-66fff2d8bedf\" (UID: \"01a3afe0-c101-49e1-ae81-66fff2d8bedf\") "
Oct 30 19:29:25.472112 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:25.472080    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01a3afe0-c101-49e1-ae81-66fff2d8bedf-maas-api-openshift-ca" pod "01a3afe0-c101-49e1-ae81-66fff2d8bedf" (UID: "01a3afe0-c101-49e1-ae81-66fff2d8bedf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:29:25.479774 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:25.477940    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01a3afe0-c101-49e1-ae81-66fff2d8bedf-kube-api-access-qmx98" pod "01a3afe0-c101-49e1-ae81-66fff2d8bedf" (UID: "01a3afe0-c101-49e1-ae81-66fff2d8bedf"). InnerVolumeSpecName "kube-api-access-qmx98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:29:25.487781 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:25.485124    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01a3afe0-c101-49e1-ae81-66fff2d8bedf-maas-api-xks-ca" pod "01a3afe0-c101-49e1-ae81-66fff2d8bedf" (UID: "01a3afe0-c101-49e1-ae81-66fff2d8bedf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:29:25.613822 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:25.613211    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmx98\" (UniqueName: \"kubernetes.io/projected/01a3afe0-c101-49e1-ae81-66fff2d8bedf-kube-api-access-qmx98\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:25.613822 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:25.613251    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01a3afe0-c101-49e1-ae81-66fff2d8bedf-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:25.613822 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:25.613267    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01a3afe0-c101-49e1-ae81-66fff2d8bedf-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:25.879815 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:25.878713    5043 generic.go:356] "Generic (PLEG): container finished" podID="b9c119ec-7866-483b-a775-e3438b62b386" containerID="5d1c5fd409794239876528ad39b0b06ef0f78b809eb7c91d1d11132305727474" exitCode=0
Oct 30 19:29:25.879815 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:25.879055    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee18dc2-cmntm" event={"ID":"b9c119ec-7866-483b-a775-e3438b62b386","Type":"ContainerDied","Data":"5d1c5fd409794239876528ad39b0b06ef0f78b809eb7c91d1d11132305727474"}
Oct 30 19:29:25.899598 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:25.881647    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d1bdfd-n8fks" event={"ID":"01a3afe0-c101-49e1-ae81-66fff2d8bedf","Type":"ContainerDied","Data":"5f83b014429623180f19cc29126c73e104cdce0f625549f7732e48fccfa2ae35"}
Oct 30 19:29:25.899598 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:25.881778    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f83b014429623180f19cc29126c73e104cdce0f625549f7732e48fccfa2ae35"
Oct 30 19:29:25.899598 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:25.881844    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66d1bdfd-n8fks"
Oct 30 19:29:27.066765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:27.066118    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:27.099764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:27.096051    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:27.217824 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:27.213845    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:27.217824 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:27.215069    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:27.240879 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:27.240828    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:27.255102 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:27.255050    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:27.272763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:27.272690    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:27.273720 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:27.273690    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:27.334940 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:27.333841    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:27.572093 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:27.571987    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:27.729788 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:27.729723    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": context deadline exceeded"
Oct 30 19:29:28.001760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:28.001266    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:29:28.304154 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:28.289881    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:28.304154 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:28.290359    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:28.304154 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:28.290752    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:29:28.304154 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:28.290899    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:29:28.339345 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:28.339294    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:28.429764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:28.394092    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:28.453762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:28.430634    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"
Oct 30 19:29:28.453762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:28.430865    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:29:28.480764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:28.455292    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:29:28.641842 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:28.474116    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:29:28.641842 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:28.474408    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:29:28.641842 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:28.626342    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:29:28.758761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:28.755048    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 30 19:29:28.987965 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:28.987931    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:29:29.046899 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:29.046847    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-6dg26" podUID="692c558e-cc23-4533-b6ad-0f982a82c187" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:29.099814 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:29.087900    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zt5b2" podUID="a348d1da-3b22-4650-84a5-4e2de27d37e3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:29.213761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:29.204786    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:29:29.302825 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:29.293868    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:29.302825 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:29.294225    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:29:29.436756 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:29.436693    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:29.439366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:29.436547    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:29:29.441493 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:29.441472    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:29:29.473679 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:29.473649    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:29:29.474202 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:29.474184    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:29:29.642866 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:29.639119    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"
Oct 30 19:29:29.642866 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:29.639169    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:29:29.642866 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:29.514037    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:29.642866 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:29.559959    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee18dc2-cmntm"
Oct 30 19:29:29.642866 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:29.637015    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:29.790073 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:29.728398    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9c119ec-7866-483b-a775-e3438b62b386-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9c119ec-7866-483b-a775-e3438b62b386-maas-api-xks-ca\") pod \"b9c119ec-7866-483b-a775-e3438b62b386\" (UID: \"b9c119ec-7866-483b-a775-e3438b62b386\") "
Oct 30 19:29:29.818180 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:29.806970    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9c119ec-7866-483b-a775-e3438b62b386-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9c119ec-7866-483b-a775-e3438b62b386-maas-api-openshift-ca\") pod \"b9c119ec-7866-483b-a775-e3438b62b386\" (UID: \"b9c119ec-7866-483b-a775-e3438b62b386\") "
Oct 30 19:29:29.818180 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:29.807275    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9c119ec-7866-483b-a775-e3438b62b386-kube-api-access-sxzxb\" (UniqueName: \"kubernetes.io/projected/b9c119ec-7866-483b-a775-e3438b62b386-kube-api-access-sxzxb\") pod \"b9c119ec-7866-483b-a775-e3438b62b386\" (UID: \"b9c119ec-7866-483b-a775-e3438b62b386\") "
Oct 30 19:29:29.823903 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:29.823856    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9c119ec-7866-483b-a775-e3438b62b386-maas-api-openshift-ca" pod "b9c119ec-7866-483b-a775-e3438b62b386" (UID: "b9c119ec-7866-483b-a775-e3438b62b386"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:29:29.844920 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:29.844874    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9c119ec-7866-483b-a775-e3438b62b386-maas-api-xks-ca" pod "b9c119ec-7866-483b-a775-e3438b62b386" (UID: "b9c119ec-7866-483b-a775-e3438b62b386"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:29:30.093821 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:29.848701    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9c119ec-7866-483b-a775-e3438b62b386-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:30.093821 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:29.848758    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9c119ec-7866-483b-a775-e3438b62b386-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:30.093821 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:30.056814    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:30.156881 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:30.156831    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:30.205762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:30.169853    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:29:30.208512 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:30.208472    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:29:30.211104 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:30.194345    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:30.211524 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:30.211499    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": context deadline exceeded"
Oct 30 19:29:30.459942 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:30.458789    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:29:30.475666 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:30.469841    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:29:30.475666 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:30.458337    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:30.484605 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:30.480532    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:30.484605 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:30.480591    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"
Oct 30 19:29:30.491295 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:30.491257    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9c119ec-7866-483b-a775-e3438b62b386-kube-api-access-sxzxb" pod "b9c119ec-7866-483b-a775-e3438b62b386" (UID: "b9c119ec-7866-483b-a775-e3438b62b386"). InnerVolumeSpecName "kube-api-access-sxzxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:29:30.509896 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:30.509851    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded"
Oct 30 19:29:30.546451 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:30.546424    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:29:30.551384 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:30.550869    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee18dc2-cmntm" event={"ID":"b9c119ec-7866-483b-a775-e3438b62b386","Type":"ContainerDied","Data":"8567521d5b0c346c36617af1e161d42d48b6279e12cae6f7cf0b1e72647899e3"}
Oct 30 19:29:30.551384 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:30.550938    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8567521d5b0c346c36617af1e161d42d48b6279e12cae6f7cf0b1e72647899e3"
Oct 30 19:29:30.551384 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:30.551001    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee18dc2-cmntm"
Oct 30 19:29:30.558314 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:30.557230    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:29:30.612760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:30.603911    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxzxb\" (UniqueName: \"kubernetes.io/projected/b9c119ec-7866-483b-a775-e3438b62b386-kube-api-access-sxzxb\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:30.627013 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:30.626984    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:29:30.627279 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:30.627262    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"
Oct 30 19:29:30.641756 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:30.637315    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:29:30.712668 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:30.712296    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:29:30.712668 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:30.712395    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:29:30.712668 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:30.712431    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:29:30.725662 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:30.725635    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:29:31.058248 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:31.058208    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d51b507-prwfs"]
Oct 30 19:29:31.059261 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:31.059238    5043 cpu_manager.go:414] "containerMap: removing container" podUID="b9c119ec-7866-483b-a775-e3438b62b386" containerName="revoke-keys"
Oct 30 19:29:31.059313 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:31.059268    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9c119ec-7866-483b-a775-e3438b62b386" containerName="revoke-keys"
Oct 30 19:29:31.059313 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:31.059287    5043 cpu_manager.go:414] "containerMap: removing container" podUID="01a3afe0-c101-49e1-ae81-66fff2d8bedf" containerName="revoke-keys"
Oct 30 19:29:31.059313 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:31.059294    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01a3afe0-c101-49e1-ae81-66fff2d8bedf" containerName="revoke-keys"
Oct 30 19:29:31.059505 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:31.059489    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="01a3afe0-c101-49e1-ae81-66fff2d8bedf" containerName="revoke-keys"
Oct 30 19:29:31.059562 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:31.059527    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9c119ec-7866-483b-a775-e3438b62b386" containerName="revoke-keys"
Oct 30 19:29:31.068844 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:31.066175    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d51b507-prwfs"
Oct 30 19:29:31.072980 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:31.072944    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:29:31.073148 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:31.073019    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:29:31.073148 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:31.073130    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:29:31.073300 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:31.073278    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:29:31.073756 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:31.073612    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d51b507-prwfs"]
Oct 30 19:29:31.226194 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:31.226158    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47cab750-d31e-4168-9817-024dc97aa9f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d51b507-prwfs\" (UID: \"47cab750-d31e-4168-9817-024dc97aa9f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d51b507-prwfs"
Oct 30 19:29:31.226391 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:31.226216    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lld5c\" (UniqueName: \"kubernetes.io/projected/47cab750-d31e-4168-9817-024dc97aa9f0-kube-api-access-lld5c\") pod \"maas-api-revoke-sub-1d51b507-prwfs\" (UID: \"47cab750-d31e-4168-9817-024dc97aa9f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d51b507-prwfs"
Oct 30 19:29:31.226511 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:31.226489    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47cab750-d31e-4168-9817-024dc97aa9f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d51b507-prwfs\" (UID: \"47cab750-d31e-4168-9817-024dc97aa9f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d51b507-prwfs"
Oct 30 19:29:31.328149 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:31.328064    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47cab750-d31e-4168-9817-024dc97aa9f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d51b507-prwfs\" (UID: \"47cab750-d31e-4168-9817-024dc97aa9f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d51b507-prwfs"
Oct 30 19:29:31.328306 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:31.328208    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47cab750-d31e-4168-9817-024dc97aa9f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d51b507-prwfs\" (UID: \"47cab750-d31e-4168-9817-024dc97aa9f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d51b507-prwfs"
Oct 30 19:29:31.328306 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:31.328273    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lld5c\" (UniqueName: \"kubernetes.io/projected/47cab750-d31e-4168-9817-024dc97aa9f0-kube-api-access-lld5c\") pod \"maas-api-revoke-sub-1d51b507-prwfs\" (UID: \"47cab750-d31e-4168-9817-024dc97aa9f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d51b507-prwfs"
Oct 30 19:29:31.328802 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:31.328778    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47cab750-d31e-4168-9817-024dc97aa9f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d51b507-prwfs\" (UID: \"47cab750-d31e-4168-9817-024dc97aa9f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d51b507-prwfs"
Oct 30 19:29:31.331107 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:31.331084    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47cab750-d31e-4168-9817-024dc97aa9f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d51b507-prwfs\" (UID: \"47cab750-d31e-4168-9817-024dc97aa9f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d51b507-prwfs"
Oct 30 19:29:31.338431 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:31.338402    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lld5c\" (UniqueName: \"kubernetes.io/projected/47cab750-d31e-4168-9817-024dc97aa9f0-kube-api-access-lld5c\") pod \"maas-api-revoke-sub-1d51b507-prwfs\" (UID: \"47cab750-d31e-4168-9817-024dc97aa9f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d51b507-prwfs"
Oct 30 19:29:31.397868 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:31.397828    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d51b507-prwfs"
Oct 30 19:29:31.714002 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:31.713973    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d51b507-prwfs"]
Oct 30 19:29:31.872579 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:31.872547    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34ac9afd-6f5lh"]
Oct 30 19:29:31.878775 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:31.878749    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ac9afd-6f5lh"
Oct 30 19:29:31.894786 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:31.888877    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34ac9afd-6f5lh"]
Oct 30 19:29:32.039800 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:32.039770    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fda28b52-1ec6-4771-b1ae-d02a0598af42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34ac9afd-6f5lh\" (UID: \"fda28b52-1ec6-4771-b1ae-d02a0598af42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ac9afd-6f5lh"
Oct 30 19:29:32.039992 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:32.039938    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fda28b52-1ec6-4771-b1ae-d02a0598af42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34ac9afd-6f5lh\" (UID: \"fda28b52-1ec6-4771-b1ae-d02a0598af42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ac9afd-6f5lh"
Oct 30 19:29:32.040047 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:32.040021    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7922\" (UniqueName: \"kubernetes.io/projected/fda28b52-1ec6-4771-b1ae-d02a0598af42-kube-api-access-h7922\") pod \"maas-api-revoke-sub-34ac9afd-6f5lh\" (UID: \"fda28b52-1ec6-4771-b1ae-d02a0598af42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ac9afd-6f5lh"
Oct 30 19:29:32.141711 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:32.141671    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fda28b52-1ec6-4771-b1ae-d02a0598af42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34ac9afd-6f5lh\" (UID: \"fda28b52-1ec6-4771-b1ae-d02a0598af42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ac9afd-6f5lh"
Oct 30 19:29:32.141903 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:32.141759    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7922\" (UniqueName: \"kubernetes.io/projected/fda28b52-1ec6-4771-b1ae-d02a0598af42-kube-api-access-h7922\") pod \"maas-api-revoke-sub-34ac9afd-6f5lh\" (UID: \"fda28b52-1ec6-4771-b1ae-d02a0598af42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ac9afd-6f5lh"
Oct 30 19:29:32.141979 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:32.141960    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fda28b52-1ec6-4771-b1ae-d02a0598af42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34ac9afd-6f5lh\" (UID: \"fda28b52-1ec6-4771-b1ae-d02a0598af42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ac9afd-6f5lh"
Oct 30 19:29:32.142412 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:32.142393    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fda28b52-1ec6-4771-b1ae-d02a0598af42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34ac9afd-6f5lh\" (UID: \"fda28b52-1ec6-4771-b1ae-d02a0598af42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ac9afd-6f5lh"
Oct 30 19:29:32.144586 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:32.144561    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fda28b52-1ec6-4771-b1ae-d02a0598af42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34ac9afd-6f5lh\" (UID: \"fda28b52-1ec6-4771-b1ae-d02a0598af42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ac9afd-6f5lh"
Oct 30 19:29:32.150808 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:32.150721    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7922\" (UniqueName: \"kubernetes.io/projected/fda28b52-1ec6-4771-b1ae-d02a0598af42-kube-api-access-h7922\") pod \"maas-api-revoke-sub-34ac9afd-6f5lh\" (UID: \"fda28b52-1ec6-4771-b1ae-d02a0598af42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ac9afd-6f5lh"
Oct 30 19:29:32.198322 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:32.198288    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ac9afd-6f5lh"
Oct 30 19:29:32.514078 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:32.513769    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34ac9afd-6f5lh"]
Oct 30 19:29:32.601243 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:32.598128    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ac9afd-6f5lh" event={"ID":"fda28b52-1ec6-4771-b1ae-d02a0598af42","Type":"ContainerStarted","Data":"641385eaba673804de011476146fdf1d0900f01a6e2de00c3faf9c16430abf61"}
Oct 30 19:29:32.601416 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:32.601289    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d51b507-prwfs" event={"ID":"47cab750-d31e-4168-9817-024dc97aa9f0","Type":"ContainerStarted","Data":"a9a5e6a0c82cd1f7e68ad57205fd17606e061133071f88bbdcbab4599150b367"}
Oct 30 19:29:32.601416 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:32.601317    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d51b507-prwfs" event={"ID":"47cab750-d31e-4168-9817-024dc97aa9f0","Type":"ContainerStarted","Data":"23b3477c8657295cdfe1d5941a97ebc062a83226e1d14a8792a8469e4de44ecc"}
Oct 30 19:29:32.618490 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:32.618433    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d51b507-prwfs" podStartSLOduration=1.618415229 podStartE2EDuration="1.618415229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:32.617466267 +0000 UTC m=+792.128526234" watchObservedRunningTime="2026-10-30 19:29:32.618415229 +0000 UTC m=+792.129475195"
Oct 30 19:29:32.790857 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:32.790824    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46b354b3-f2tds"]
Oct 30 19:29:32.796372 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:32.796345    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46b354b3-f2tds"
Oct 30 19:29:32.803612 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:32.803579    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46b354b3-f2tds"]
Oct 30 19:29:32.954533 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:32.954501    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db1bb288-ae84-4f59-bf73-2152e976377b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46b354b3-f2tds\" (UID: \"db1bb288-ae84-4f59-bf73-2152e976377b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46b354b3-f2tds"
Oct 30 19:29:32.954533 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:32.954536    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s279x\" (UniqueName: \"kubernetes.io/projected/db1bb288-ae84-4f59-bf73-2152e976377b-kube-api-access-s279x\") pod \"maas-api-revoke-sub-46b354b3-f2tds\" (UID: \"db1bb288-ae84-4f59-bf73-2152e976377b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46b354b3-f2tds"
Oct 30 19:29:32.954775 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:32.954692    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db1bb288-ae84-4f59-bf73-2152e976377b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46b354b3-f2tds\" (UID: \"db1bb288-ae84-4f59-bf73-2152e976377b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46b354b3-f2tds"
Oct 30 19:29:33.055963 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:33.055863    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db1bb288-ae84-4f59-bf73-2152e976377b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46b354b3-f2tds\" (UID: \"db1bb288-ae84-4f59-bf73-2152e976377b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46b354b3-f2tds"
Oct 30 19:29:33.055963 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:33.055903    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s279x\" (UniqueName: \"kubernetes.io/projected/db1bb288-ae84-4f59-bf73-2152e976377b-kube-api-access-s279x\") pod \"maas-api-revoke-sub-46b354b3-f2tds\" (UID: \"db1bb288-ae84-4f59-bf73-2152e976377b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46b354b3-f2tds"
Oct 30 19:29:33.056201 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:33.055995    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db1bb288-ae84-4f59-bf73-2152e976377b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46b354b3-f2tds\" (UID: \"db1bb288-ae84-4f59-bf73-2152e976377b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46b354b3-f2tds"
Oct 30 19:29:33.057229 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:33.057196    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db1bb288-ae84-4f59-bf73-2152e976377b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46b354b3-f2tds\" (UID: \"db1bb288-ae84-4f59-bf73-2152e976377b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46b354b3-f2tds"
Oct 30 19:29:33.058931 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:33.058908    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db1bb288-ae84-4f59-bf73-2152e976377b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46b354b3-f2tds\" (UID: \"db1bb288-ae84-4f59-bf73-2152e976377b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46b354b3-f2tds"
Oct 30 19:29:33.106494 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:33.106449    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s279x\" (UniqueName: \"kubernetes.io/projected/db1bb288-ae84-4f59-bf73-2152e976377b-kube-api-access-s279x\") pod \"maas-api-revoke-sub-46b354b3-f2tds\" (UID: \"db1bb288-ae84-4f59-bf73-2152e976377b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46b354b3-f2tds"
Oct 30 19:29:33.154856 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:33.154817    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46b354b3-f2tds"
Oct 30 19:29:33.469326 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:29:33.469290    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb1bb288_ae84_4f59_bf73_2152e976377b.slice/crio-cf5e9adfbc2ce1576874c395b5111ab006e1d52041bf25d5fb069b2547705ac0 WatchSource:0}: Error finding container cf5e9adfbc2ce1576874c395b5111ab006e1d52041bf25d5fb069b2547705ac0: Status 404 returned error can't find the container with id cf5e9adfbc2ce1576874c395b5111ab006e1d52041bf25d5fb069b2547705ac0
Oct 30 19:29:33.474160 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:33.474130    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46b354b3-f2tds"]
Oct 30 19:29:33.612220 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:33.612188    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ac9afd-6f5lh" event={"ID":"fda28b52-1ec6-4771-b1ae-d02a0598af42","Type":"ContainerStarted","Data":"412b9e685ac280b1a51d4da0545707f9c83b91ef14ae6d2d6a6d8f50e1f773ea"}
Oct 30 19:29:33.613988 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:33.613965    5043 generic.go:356] "Generic (PLEG): container finished" podID="47cab750-d31e-4168-9817-024dc97aa9f0" containerID="a9a5e6a0c82cd1f7e68ad57205fd17606e061133071f88bbdcbab4599150b367" exitCode=0
Oct 30 19:29:33.614111 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:33.614039    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d51b507-prwfs" event={"ID":"47cab750-d31e-4168-9817-024dc97aa9f0","Type":"ContainerDied","Data":"a9a5e6a0c82cd1f7e68ad57205fd17606e061133071f88bbdcbab4599150b367"}
Oct 30 19:29:33.615354 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:33.615335    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46b354b3-f2tds" event={"ID":"db1bb288-ae84-4f59-bf73-2152e976377b","Type":"ContainerStarted","Data":"cf5e9adfbc2ce1576874c395b5111ab006e1d52041bf25d5fb069b2547705ac0"}
Oct 30 19:29:33.634870 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:33.634823    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ac9afd-6f5lh" podStartSLOduration=2.634810818 podStartE2EDuration="2.634810818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:33.630377822 +0000 UTC m=+793.141437788" watchObservedRunningTime="2026-10-30 19:29:33.634810818 +0000 UTC m=+793.145870783"
Oct 30 19:29:34.626064 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:34.626034    5043 generic.go:356] "Generic (PLEG): container finished" podID="fda28b52-1ec6-4771-b1ae-d02a0598af42" containerID="412b9e685ac280b1a51d4da0545707f9c83b91ef14ae6d2d6a6d8f50e1f773ea" exitCode=0
Oct 30 19:29:34.626502 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:34.626116    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ac9afd-6f5lh" event={"ID":"fda28b52-1ec6-4771-b1ae-d02a0598af42","Type":"ContainerDied","Data":"412b9e685ac280b1a51d4da0545707f9c83b91ef14ae6d2d6a6d8f50e1f773ea"}
Oct 30 19:29:34.627985 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:34.627961    5043 generic.go:356] "Generic (PLEG): container finished" podID="db1bb288-ae84-4f59-bf73-2152e976377b" containerID="cc85642a61c9eda036c2c852e0dd56906e127028bc6425b6aa4c5b10d0abf88c" exitCode=0
Oct 30 19:29:34.628108 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:34.628047    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46b354b3-f2tds" event={"ID":"db1bb288-ae84-4f59-bf73-2152e976377b","Type":"ContainerDied","Data":"cc85642a61c9eda036c2c852e0dd56906e127028bc6425b6aa4c5b10d0abf88c"}
Oct 30 19:29:34.818849 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:34.818821    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d51b507-prwfs"
Oct 30 19:29:34.994483 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:34.994452    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47cab750-d31e-4168-9817-024dc97aa9f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47cab750-d31e-4168-9817-024dc97aa9f0-maas-api-openshift-ca\") pod \"47cab750-d31e-4168-9817-024dc97aa9f0\" (UID: \"47cab750-d31e-4168-9817-024dc97aa9f0\") "
Oct 30 19:29:34.994691 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:34.994578    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47cab750-d31e-4168-9817-024dc97aa9f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47cab750-d31e-4168-9817-024dc97aa9f0-maas-api-xks-ca\") pod \"47cab750-d31e-4168-9817-024dc97aa9f0\" (UID: \"47cab750-d31e-4168-9817-024dc97aa9f0\") "
Oct 30 19:29:34.994786 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:34.994767    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47cab750-d31e-4168-9817-024dc97aa9f0-kube-api-access-lld5c\" (UniqueName: \"kubernetes.io/projected/47cab750-d31e-4168-9817-024dc97aa9f0-kube-api-access-lld5c\") pod \"47cab750-d31e-4168-9817-024dc97aa9f0\" (UID: \"47cab750-d31e-4168-9817-024dc97aa9f0\") "
Oct 30 19:29:34.994913 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:34.994891    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47cab750-d31e-4168-9817-024dc97aa9f0-maas-api-openshift-ca" pod "47cab750-d31e-4168-9817-024dc97aa9f0" (UID: "47cab750-d31e-4168-9817-024dc97aa9f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:29:34.995326 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:34.995306    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47cab750-d31e-4168-9817-024dc97aa9f0-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:34.997561 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:34.997533    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47cab750-d31e-4168-9817-024dc97aa9f0-maas-api-xks-ca" pod "47cab750-d31e-4168-9817-024dc97aa9f0" (UID: "47cab750-d31e-4168-9817-024dc97aa9f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:29:34.997674 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:34.997579    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47cab750-d31e-4168-9817-024dc97aa9f0-kube-api-access-lld5c" pod "47cab750-d31e-4168-9817-024dc97aa9f0" (UID: "47cab750-d31e-4168-9817-024dc97aa9f0"). InnerVolumeSpecName "kube-api-access-lld5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:29:35.096590 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:35.096555    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47cab750-d31e-4168-9817-024dc97aa9f0-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:35.096590 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:35.096585    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lld5c\" (UniqueName: \"kubernetes.io/projected/47cab750-d31e-4168-9817-024dc97aa9f0-kube-api-access-lld5c\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:35.453427 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:35.453338    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00c784c1-7rgrb"]
Oct 30 19:29:35.454044 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:35.454024    5043 cpu_manager.go:414] "containerMap: removing container" podUID="47cab750-d31e-4168-9817-024dc97aa9f0" containerName="revoke-keys"
Oct 30 19:29:35.454044 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:35.454042    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47cab750-d31e-4168-9817-024dc97aa9f0" containerName="revoke-keys"
Oct 30 19:29:35.454196 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:35.454186    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="47cab750-d31e-4168-9817-024dc97aa9f0" containerName="revoke-keys"
Oct 30 19:29:35.458913 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:35.458698    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c784c1-7rgrb"
Oct 30 19:29:35.490955 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:35.490917    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00c784c1-7rgrb"]
Oct 30 19:29:35.606941 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:35.606817    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pfqv\" (UniqueName: \"kubernetes.io/projected/34ada6b0-b808-4f1d-a770-4ce3219722ca-kube-api-access-9pfqv\") pod \"maas-api-revoke-sub-00c784c1-7rgrb\" (UID: \"34ada6b0-b808-4f1d-a770-4ce3219722ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c784c1-7rgrb"
Oct 30 19:29:35.607126 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:35.606998    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34ada6b0-b808-4f1d-a770-4ce3219722ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00c784c1-7rgrb\" (UID: \"34ada6b0-b808-4f1d-a770-4ce3219722ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c784c1-7rgrb"
Oct 30 19:29:35.607211 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:35.607179    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34ada6b0-b808-4f1d-a770-4ce3219722ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00c784c1-7rgrb\" (UID: \"34ada6b0-b808-4f1d-a770-4ce3219722ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c784c1-7rgrb"
Oct 30 19:29:35.639446 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:35.639419    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d51b507-prwfs"
Oct 30 19:29:35.639446 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:35.639413    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d51b507-prwfs" event={"ID":"47cab750-d31e-4168-9817-024dc97aa9f0","Type":"ContainerDied","Data":"23b3477c8657295cdfe1d5941a97ebc062a83226e1d14a8792a8469e4de44ecc"}
Oct 30 19:29:35.639927 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:35.639468    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23b3477c8657295cdfe1d5941a97ebc062a83226e1d14a8792a8469e4de44ecc"
Oct 30 19:29:35.709510 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:35.708826    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pfqv\" (UniqueName: \"kubernetes.io/projected/34ada6b0-b808-4f1d-a770-4ce3219722ca-kube-api-access-9pfqv\") pod \"maas-api-revoke-sub-00c784c1-7rgrb\" (UID: \"34ada6b0-b808-4f1d-a770-4ce3219722ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c784c1-7rgrb"
Oct 30 19:29:35.709510 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:35.708910    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34ada6b0-b808-4f1d-a770-4ce3219722ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00c784c1-7rgrb\" (UID: \"34ada6b0-b808-4f1d-a770-4ce3219722ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c784c1-7rgrb"
Oct 30 19:29:35.726392 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:35.710288    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34ada6b0-b808-4f1d-a770-4ce3219722ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00c784c1-7rgrb\" (UID: \"34ada6b0-b808-4f1d-a770-4ce3219722ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c784c1-7rgrb"
Oct 30 19:29:35.726392 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:35.716824    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34ada6b0-b808-4f1d-a770-4ce3219722ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00c784c1-7rgrb\" (UID: \"34ada6b0-b808-4f1d-a770-4ce3219722ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c784c1-7rgrb"
Oct 30 19:29:35.727947 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:35.726724    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pfqv\" (UniqueName: \"kubernetes.io/projected/34ada6b0-b808-4f1d-a770-4ce3219722ca-kube-api-access-9pfqv\") pod \"maas-api-revoke-sub-00c784c1-7rgrb\" (UID: \"34ada6b0-b808-4f1d-a770-4ce3219722ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c784c1-7rgrb"
Oct 30 19:29:35.727947 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:35.727665    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34ada6b0-b808-4f1d-a770-4ce3219722ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00c784c1-7rgrb\" (UID: \"34ada6b0-b808-4f1d-a770-4ce3219722ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c784c1-7rgrb"
Oct 30 19:29:35.819952 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:35.815805    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c784c1-7rgrb"
Oct 30 19:29:35.836854 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:35.836825    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ac9afd-6f5lh"
Oct 30 19:29:35.876288 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:35.876258    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46b354b3-f2tds"
Oct 30 19:29:36.015011 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:36.014973    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db1bb288-ae84-4f59-bf73-2152e976377b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db1bb288-ae84-4f59-bf73-2152e976377b-maas-api-openshift-ca\") pod \"db1bb288-ae84-4f59-bf73-2152e976377b\" (UID: \"db1bb288-ae84-4f59-bf73-2152e976377b\") "
Oct 30 19:29:36.015192 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:36.015068    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db1bb288-ae84-4f59-bf73-2152e976377b-kube-api-access-s279x\" (UniqueName: \"kubernetes.io/projected/db1bb288-ae84-4f59-bf73-2152e976377b-kube-api-access-s279x\") pod \"db1bb288-ae84-4f59-bf73-2152e976377b\" (UID: \"db1bb288-ae84-4f59-bf73-2152e976377b\") "
Oct 30 19:29:36.015283 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:36.015190    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fda28b52-1ec6-4771-b1ae-d02a0598af42-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fda28b52-1ec6-4771-b1ae-d02a0598af42-maas-api-xks-ca\") pod \"fda28b52-1ec6-4771-b1ae-d02a0598af42\" (UID: \"fda28b52-1ec6-4771-b1ae-d02a0598af42\") "
Oct 30 19:29:36.015283 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:36.015224    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db1bb288-ae84-4f59-bf73-2152e976377b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db1bb288-ae84-4f59-bf73-2152e976377b-maas-api-xks-ca\") pod \"db1bb288-ae84-4f59-bf73-2152e976377b\" (UID: \"db1bb288-ae84-4f59-bf73-2152e976377b\") "
Oct 30 19:29:36.015388 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:36.015322    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fda28b52-1ec6-4771-b1ae-d02a0598af42-kube-api-access-h7922\" (UniqueName: \"kubernetes.io/projected/fda28b52-1ec6-4771-b1ae-d02a0598af42-kube-api-access-h7922\") pod \"fda28b52-1ec6-4771-b1ae-d02a0598af42\" (UID: \"fda28b52-1ec6-4771-b1ae-d02a0598af42\") "
Oct 30 19:29:36.015442 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:36.015411    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db1bb288-ae84-4f59-bf73-2152e976377b-maas-api-openshift-ca" pod "db1bb288-ae84-4f59-bf73-2152e976377b" (UID: "db1bb288-ae84-4f59-bf73-2152e976377b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:29:36.015603 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:36.015583    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fda28b52-1ec6-4771-b1ae-d02a0598af42-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fda28b52-1ec6-4771-b1ae-d02a0598af42-maas-api-openshift-ca\") pod \"fda28b52-1ec6-4771-b1ae-d02a0598af42\" (UID: \"fda28b52-1ec6-4771-b1ae-d02a0598af42\") "
Oct 30 19:29:36.016006 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:36.015972    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fda28b52-1ec6-4771-b1ae-d02a0598af42-maas-api-openshift-ca" pod "fda28b52-1ec6-4771-b1ae-d02a0598af42" (UID: "fda28b52-1ec6-4771-b1ae-d02a0598af42"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:29:36.016428 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:36.016384    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fda28b52-1ec6-4771-b1ae-d02a0598af42-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:36.016504 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:36.016436    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db1bb288-ae84-4f59-bf73-2152e976377b-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:36.017892 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:36.017868    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db1bb288-ae84-4f59-bf73-2152e976377b-maas-api-xks-ca" pod "db1bb288-ae84-4f59-bf73-2152e976377b" (UID: "db1bb288-ae84-4f59-bf73-2152e976377b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:29:36.017996 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:36.017982    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fda28b52-1ec6-4771-b1ae-d02a0598af42-kube-api-access-h7922" pod "fda28b52-1ec6-4771-b1ae-d02a0598af42" (UID: "fda28b52-1ec6-4771-b1ae-d02a0598af42"). InnerVolumeSpecName "kube-api-access-h7922". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:29:36.018247 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:36.018228    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db1bb288-ae84-4f59-bf73-2152e976377b-kube-api-access-s279x" pod "db1bb288-ae84-4f59-bf73-2152e976377b" (UID: "db1bb288-ae84-4f59-bf73-2152e976377b"). InnerVolumeSpecName "kube-api-access-s279x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:29:36.018382 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:36.018360    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fda28b52-1ec6-4771-b1ae-d02a0598af42-maas-api-xks-ca" pod "fda28b52-1ec6-4771-b1ae-d02a0598af42" (UID: "fda28b52-1ec6-4771-b1ae-d02a0598af42"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:29:36.117667 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:36.117622    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7922\" (UniqueName: \"kubernetes.io/projected/fda28b52-1ec6-4771-b1ae-d02a0598af42-kube-api-access-h7922\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:36.117667 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:36.117664    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s279x\" (UniqueName: \"kubernetes.io/projected/db1bb288-ae84-4f59-bf73-2152e976377b-kube-api-access-s279x\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:36.117918 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:36.117682    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fda28b52-1ec6-4771-b1ae-d02a0598af42-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:36.117918 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:36.117698    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db1bb288-ae84-4f59-bf73-2152e976377b-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:36.142036 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:29:36.142007    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34ada6b0_b808_4f1d_a770_4ce3219722ca.slice/crio-92fd913bff42f2f724443a00376a26921a6e06b0d98d59bc2136701fbf38d13b WatchSource:0}: Error finding container 92fd913bff42f2f724443a00376a26921a6e06b0d98d59bc2136701fbf38d13b: Status 404 returned error can't find the container with id 92fd913bff42f2f724443a00376a26921a6e06b0d98d59bc2136701fbf38d13b
Oct 30 19:29:36.142318 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:36.142197    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00c784c1-7rgrb"]
Oct 30 19:29:36.653100 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:36.652999    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c784c1-7rgrb" event={"ID":"34ada6b0-b808-4f1d-a770-4ce3219722ca","Type":"ContainerStarted","Data":"36253c507721b59c5eeeef3086d50b1e3e2424f39eaef2e2eca94ba72696dc85"}
Oct 30 19:29:36.653100 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:36.653062    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c784c1-7rgrb" event={"ID":"34ada6b0-b808-4f1d-a770-4ce3219722ca","Type":"ContainerStarted","Data":"92fd913bff42f2f724443a00376a26921a6e06b0d98d59bc2136701fbf38d13b"}
Oct 30 19:29:36.653709 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:36.653474    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ac9afd-6f5lh"
Oct 30 19:29:36.653709 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:36.653524    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ac9afd-6f5lh" event={"ID":"fda28b52-1ec6-4771-b1ae-d02a0598af42","Type":"ContainerDied","Data":"641385eaba673804de011476146fdf1d0900f01a6e2de00c3faf9c16430abf61"}
Oct 30 19:29:36.653709 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:36.653572    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="641385eaba673804de011476146fdf1d0900f01a6e2de00c3faf9c16430abf61"
Oct 30 19:29:36.661985 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:36.661955    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46b354b3-f2tds"
Oct 30 19:29:36.662136 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:36.661987    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46b354b3-f2tds" event={"ID":"db1bb288-ae84-4f59-bf73-2152e976377b","Type":"ContainerDied","Data":"cf5e9adfbc2ce1576874c395b5111ab006e1d52041bf25d5fb069b2547705ac0"}
Oct 30 19:29:36.662136 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:36.662030    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf5e9adfbc2ce1576874c395b5111ab006e1d52041bf25d5fb069b2547705ac0"
Oct 30 19:29:36.680823 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:36.680775    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c784c1-7rgrb" podStartSLOduration=1.680761846 podStartE2EDuration="1.680761846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:36.672481308 +0000 UTC m=+796.183541274" watchObservedRunningTime="2026-10-30 19:29:36.680761846 +0000 UTC m=+796.191821805"
Oct 30 19:29:37.677185 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:37.677145    5043 generic.go:356] "Generic (PLEG): container finished" podID="34ada6b0-b808-4f1d-a770-4ce3219722ca" containerID="36253c507721b59c5eeeef3086d50b1e3e2424f39eaef2e2eca94ba72696dc85" exitCode=0
Oct 30 19:29:37.678131 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:37.678102    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c784c1-7rgrb" event={"ID":"34ada6b0-b808-4f1d-a770-4ce3219722ca","Type":"ContainerDied","Data":"36253c507721b59c5eeeef3086d50b1e3e2424f39eaef2e2eca94ba72696dc85"}
Oct 30 19:29:38.176905 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:38.176867    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-799546ec-lnflr"]
Oct 30 19:29:38.188143 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:38.188104    5043 cpu_manager.go:414] "containerMap: removing container" podUID="db1bb288-ae84-4f59-bf73-2152e976377b" containerName="revoke-keys"
Oct 30 19:29:38.188143 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:38.188142    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db1bb288-ae84-4f59-bf73-2152e976377b" containerName="revoke-keys"
Oct 30 19:29:38.188421 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:38.188287    5043 cpu_manager.go:414] "containerMap: removing container" podUID="fda28b52-1ec6-4771-b1ae-d02a0598af42" containerName="revoke-keys"
Oct 30 19:29:38.188421 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:38.188300    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fda28b52-1ec6-4771-b1ae-d02a0598af42" containerName="revoke-keys"
Oct 30 19:29:38.188543 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:38.188500    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="fda28b52-1ec6-4771-b1ae-d02a0598af42" containerName="revoke-keys"
Oct 30 19:29:38.188543 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:38.188517    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="db1bb288-ae84-4f59-bf73-2152e976377b" containerName="revoke-keys"
Oct 30 19:29:38.193808 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:38.193783    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-799546ec-lnflr"
Oct 30 19:29:38.197529 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:38.197327    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-799546ec-lnflr"]
Oct 30 19:29:38.248690 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:38.248648    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2xpz\" (UniqueName: \"kubernetes.io/projected/51267af2-5e1b-438e-b1be-25f8f5ee2d9f-kube-api-access-z2xpz\") pod \"maas-api-revoke-sub-799546ec-lnflr\" (UID: \"51267af2-5e1b-438e-b1be-25f8f5ee2d9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-799546ec-lnflr"
Oct 30 19:29:38.248967 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:38.248802    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51267af2-5e1b-438e-b1be-25f8f5ee2d9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-799546ec-lnflr\" (UID: \"51267af2-5e1b-438e-b1be-25f8f5ee2d9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-799546ec-lnflr"
Oct 30 19:29:38.248967 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:38.248925    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51267af2-5e1b-438e-b1be-25f8f5ee2d9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-799546ec-lnflr\" (UID: \"51267af2-5e1b-438e-b1be-25f8f5ee2d9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-799546ec-lnflr"
Oct 30 19:29:38.349960 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:38.349915    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2xpz\" (UniqueName: \"kubernetes.io/projected/51267af2-5e1b-438e-b1be-25f8f5ee2d9f-kube-api-access-z2xpz\") pod \"maas-api-revoke-sub-799546ec-lnflr\" (UID: \"51267af2-5e1b-438e-b1be-25f8f5ee2d9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-799546ec-lnflr"
Oct 30 19:29:38.350199 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:38.350019    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51267af2-5e1b-438e-b1be-25f8f5ee2d9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-799546ec-lnflr\" (UID: \"51267af2-5e1b-438e-b1be-25f8f5ee2d9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-799546ec-lnflr"
Oct 30 19:29:38.350199 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:38.350106    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51267af2-5e1b-438e-b1be-25f8f5ee2d9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-799546ec-lnflr\" (UID: \"51267af2-5e1b-438e-b1be-25f8f5ee2d9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-799546ec-lnflr"
Oct 30 19:29:38.350754 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:38.350718    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51267af2-5e1b-438e-b1be-25f8f5ee2d9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-799546ec-lnflr\" (UID: \"51267af2-5e1b-438e-b1be-25f8f5ee2d9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-799546ec-lnflr"
Oct 30 19:29:38.352685 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:38.352658    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51267af2-5e1b-438e-b1be-25f8f5ee2d9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-799546ec-lnflr\" (UID: \"51267af2-5e1b-438e-b1be-25f8f5ee2d9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-799546ec-lnflr"
Oct 30 19:29:38.359062 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:38.359034    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2xpz\" (UniqueName: \"kubernetes.io/projected/51267af2-5e1b-438e-b1be-25f8f5ee2d9f-kube-api-access-z2xpz\") pod \"maas-api-revoke-sub-799546ec-lnflr\" (UID: \"51267af2-5e1b-438e-b1be-25f8f5ee2d9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-799546ec-lnflr"
Oct 30 19:29:38.543368 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:38.543333    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-799546ec-lnflr"
Oct 30 19:29:38.852970 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:38.851407    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-799546ec-lnflr"]
Oct 30 19:29:38.911787 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:38.911709    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c784c1-7rgrb"
Oct 30 19:29:38.960102 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:38.960064    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34ada6b0-b808-4f1d-a770-4ce3219722ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34ada6b0-b808-4f1d-a770-4ce3219722ca-maas-api-openshift-ca\") pod \"34ada6b0-b808-4f1d-a770-4ce3219722ca\" (UID: \"34ada6b0-b808-4f1d-a770-4ce3219722ca\") "
Oct 30 19:29:38.960275 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:38.960180    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34ada6b0-b808-4f1d-a770-4ce3219722ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34ada6b0-b808-4f1d-a770-4ce3219722ca-maas-api-xks-ca\") pod \"34ada6b0-b808-4f1d-a770-4ce3219722ca\" (UID: \"34ada6b0-b808-4f1d-a770-4ce3219722ca\") "
Oct 30 19:29:38.960378 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:38.960270    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34ada6b0-b808-4f1d-a770-4ce3219722ca-kube-api-access-9pfqv\" (UniqueName: \"kubernetes.io/projected/34ada6b0-b808-4f1d-a770-4ce3219722ca-kube-api-access-9pfqv\") pod \"34ada6b0-b808-4f1d-a770-4ce3219722ca\" (UID: \"34ada6b0-b808-4f1d-a770-4ce3219722ca\") "
Oct 30 19:29:38.961563 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:38.961520    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34ada6b0-b808-4f1d-a770-4ce3219722ca-maas-api-openshift-ca" pod "34ada6b0-b808-4f1d-a770-4ce3219722ca" (UID: "34ada6b0-b808-4f1d-a770-4ce3219722ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:29:39.056637 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:39.056597    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34ada6b0-b808-4f1d-a770-4ce3219722ca-kube-api-access-9pfqv" pod "34ada6b0-b808-4f1d-a770-4ce3219722ca" (UID: "34ada6b0-b808-4f1d-a770-4ce3219722ca"). InnerVolumeSpecName "kube-api-access-9pfqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:29:39.056637 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:39.056592    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34ada6b0-b808-4f1d-a770-4ce3219722ca-maas-api-xks-ca" pod "34ada6b0-b808-4f1d-a770-4ce3219722ca" (UID: "34ada6b0-b808-4f1d-a770-4ce3219722ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:29:39.062147 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:39.062117    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pfqv\" (UniqueName: \"kubernetes.io/projected/34ada6b0-b808-4f1d-a770-4ce3219722ca-kube-api-access-9pfqv\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:39.062233 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:39.062153    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34ada6b0-b808-4f1d-a770-4ce3219722ca-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:39.062233 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:39.062171    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34ada6b0-b808-4f1d-a770-4ce3219722ca-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:39.697193 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:39.697163    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c784c1-7rgrb"
Oct 30 19:29:39.697360 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:39.697202    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c784c1-7rgrb" event={"ID":"34ada6b0-b808-4f1d-a770-4ce3219722ca","Type":"ContainerDied","Data":"92fd913bff42f2f724443a00376a26921a6e06b0d98d59bc2136701fbf38d13b"}
Oct 30 19:29:39.697360 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:39.697245    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92fd913bff42f2f724443a00376a26921a6e06b0d98d59bc2136701fbf38d13b"
Oct 30 19:29:39.699137 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:39.699110    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-799546ec-lnflr" event={"ID":"51267af2-5e1b-438e-b1be-25f8f5ee2d9f","Type":"ContainerStarted","Data":"fa86e48bb627f66aa0fcde2d3507be7bbe3562f383c72591c651f7f540706172"}
Oct 30 19:29:39.699248 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:39.699160    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-799546ec-lnflr" event={"ID":"51267af2-5e1b-438e-b1be-25f8f5ee2d9f","Type":"ContainerStarted","Data":"3a8f687eabab5a5e36e36f2954069a139f65ef9e9ba224ded6aa85a5222c7685"}
Oct 30 19:29:39.721236 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:39.721176    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-799546ec-lnflr" podStartSLOduration=1.721156605 podStartE2EDuration="1.721156605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:39.714350738 +0000 UTC m=+799.225410704" watchObservedRunningTime="2026-10-30 19:29:39.721156605 +0000 UTC m=+799.232216569"
Oct 30 19:29:40.716167 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:40.716116    5043 generic.go:356] "Generic (PLEG): container finished" podID="51267af2-5e1b-438e-b1be-25f8f5ee2d9f" containerID="fa86e48bb627f66aa0fcde2d3507be7bbe3562f383c72591c651f7f540706172" exitCode=0
Oct 30 19:29:40.716625 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:40.716192    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-799546ec-lnflr" event={"ID":"51267af2-5e1b-438e-b1be-25f8f5ee2d9f","Type":"ContainerDied","Data":"fa86e48bb627f66aa0fcde2d3507be7bbe3562f383c72591c651f7f540706172"}
Oct 30 19:29:41.277225 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.277192    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2b9fef5-vfw6s"]
Oct 30 19:29:41.277852 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.277837    5043 cpu_manager.go:414] "containerMap: removing container" podUID="34ada6b0-b808-4f1d-a770-4ce3219722ca" containerName="revoke-keys"
Oct 30 19:29:41.277852 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.277851    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34ada6b0-b808-4f1d-a770-4ce3219722ca" containerName="revoke-keys"
Oct 30 19:29:41.277999 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.277964    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="34ada6b0-b808-4f1d-a770-4ce3219722ca" containerName="revoke-keys"
Oct 30 19:29:41.281779 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.281730    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b9fef5-vfw6s"
Oct 30 19:29:41.290362 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.290330    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2b9fef5-vfw6s"]
Oct 30 19:29:41.294292 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.294256    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss8hs\" (UniqueName: \"kubernetes.io/projected/ad59b8d8-2911-4fe8-b7cd-9090ff58590c-kube-api-access-ss8hs\") pod \"maas-api-revoke-sub-c2b9fef5-vfw6s\" (UID: \"ad59b8d8-2911-4fe8-b7cd-9090ff58590c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b9fef5-vfw6s"
Oct 30 19:29:41.294431 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.294306    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad59b8d8-2911-4fe8-b7cd-9090ff58590c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2b9fef5-vfw6s\" (UID: \"ad59b8d8-2911-4fe8-b7cd-9090ff58590c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b9fef5-vfw6s"
Oct 30 19:29:41.294431 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.294397    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad59b8d8-2911-4fe8-b7cd-9090ff58590c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2b9fef5-vfw6s\" (UID: \"ad59b8d8-2911-4fe8-b7cd-9090ff58590c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b9fef5-vfw6s"
Oct 30 19:29:41.396090 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.396051    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad59b8d8-2911-4fe8-b7cd-9090ff58590c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2b9fef5-vfw6s\" (UID: \"ad59b8d8-2911-4fe8-b7cd-9090ff58590c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b9fef5-vfw6s"
Oct 30 19:29:41.396270 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.396228    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss8hs\" (UniqueName: \"kubernetes.io/projected/ad59b8d8-2911-4fe8-b7cd-9090ff58590c-kube-api-access-ss8hs\") pod \"maas-api-revoke-sub-c2b9fef5-vfw6s\" (UID: \"ad59b8d8-2911-4fe8-b7cd-9090ff58590c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b9fef5-vfw6s"
Oct 30 19:29:41.396270 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.396252    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad59b8d8-2911-4fe8-b7cd-9090ff58590c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2b9fef5-vfw6s\" (UID: \"ad59b8d8-2911-4fe8-b7cd-9090ff58590c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b9fef5-vfw6s"
Oct 30 19:29:41.396864 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.396837    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad59b8d8-2911-4fe8-b7cd-9090ff58590c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2b9fef5-vfw6s\" (UID: \"ad59b8d8-2911-4fe8-b7cd-9090ff58590c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b9fef5-vfw6s"
Oct 30 19:29:41.398858 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.398837    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad59b8d8-2911-4fe8-b7cd-9090ff58590c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2b9fef5-vfw6s\" (UID: \"ad59b8d8-2911-4fe8-b7cd-9090ff58590c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b9fef5-vfw6s"
Oct 30 19:29:41.406431 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.405270    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss8hs\" (UniqueName: \"kubernetes.io/projected/ad59b8d8-2911-4fe8-b7cd-9090ff58590c-kube-api-access-ss8hs\") pod \"maas-api-revoke-sub-c2b9fef5-vfw6s\" (UID: \"ad59b8d8-2911-4fe8-b7cd-9090ff58590c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b9fef5-vfw6s"
Oct 30 19:29:41.578461 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.578378    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q"]
Oct 30 19:29:41.584376 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.584343    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q"
Oct 30 19:29:41.599018 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.598848    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q"]
Oct 30 19:29:41.599018 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.598883    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8a7200d-6acf-4008-8156-d9102730ac76-istiod-ca-cert\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q\" (UID: \"d8a7200d-6acf-4008-8156-d9102730ac76\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q"
Oct 30 19:29:41.599018 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.598959    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8a7200d-6acf-4008-8156-d9102730ac76-istio-data\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q\" (UID: \"d8a7200d-6acf-4008-8156-d9102730ac76\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q"
Oct 30 19:29:41.599617 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.599073    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8a7200d-6acf-4008-8156-d9102730ac76-istio-podinfo\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q\" (UID: \"d8a7200d-6acf-4008-8156-d9102730ac76\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q"
Oct 30 19:29:41.599617 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.599116    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a7200d-6acf-4008-8156-d9102730ac76-workload-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q\" (UID: \"d8a7200d-6acf-4008-8156-d9102730ac76\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q"
Oct 30 19:29:41.599617 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.599264    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8a7200d-6acf-4008-8156-d9102730ac76-istio-envoy\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q\" (UID: \"d8a7200d-6acf-4008-8156-d9102730ac76\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q"
Oct 30 19:29:41.599617 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.599301    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8a7200d-6acf-4008-8156-d9102730ac76-workload-certs\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q\" (UID: \"d8a7200d-6acf-4008-8156-d9102730ac76\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q"
Oct 30 19:29:41.599617 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.599369    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8a7200d-6acf-4008-8156-d9102730ac76-istio-token\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q\" (UID: \"d8a7200d-6acf-4008-8156-d9102730ac76\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q"
Oct 30 19:29:41.599617 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.599486    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a7200d-6acf-4008-8156-d9102730ac76-credential-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q\" (UID: \"d8a7200d-6acf-4008-8156-d9102730ac76\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q"
Oct 30 19:29:41.599617 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.599534    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h9j8\" (UniqueName: \"kubernetes.io/projected/d8a7200d-6acf-4008-8156-d9102730ac76-kube-api-access-7h9j8\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q\" (UID: \"d8a7200d-6acf-4008-8156-d9102730ac76\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q"
Oct 30 19:29:41.622800 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.617173    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b9fef5-vfw6s"
Oct 30 19:29:41.652185 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.652153    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb"]
Oct 30 19:29:41.660187 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.660130    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb"
Oct 30 19:29:41.678266 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.677602    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb"]
Oct 30 19:29:41.701538 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.701249    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8a7200d-6acf-4008-8156-d9102730ac76-istiod-ca-cert\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q\" (UID: \"d8a7200d-6acf-4008-8156-d9102730ac76\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q"
Oct 30 19:29:41.701538 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.701315    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43b74d6-8c5b-430d-bffb-a30beb6996ec-credential-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb\" (UID: \"f43b74d6-8c5b-430d-bffb-a30beb6996ec\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb"
Oct 30 19:29:41.701538 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.701365    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f43b74d6-8c5b-430d-bffb-a30beb6996ec-istio-podinfo\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb\" (UID: \"f43b74d6-8c5b-430d-bffb-a30beb6996ec\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb"
Oct 30 19:29:41.701538 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.701406    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8a7200d-6acf-4008-8156-d9102730ac76-istio-data\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q\" (UID: \"d8a7200d-6acf-4008-8156-d9102730ac76\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q"
Oct 30 19:29:41.701538 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.701434    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f43b74d6-8c5b-430d-bffb-a30beb6996ec-istio-token\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb\" (UID: \"f43b74d6-8c5b-430d-bffb-a30beb6996ec\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb"
Oct 30 19:29:41.701538 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.701488    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8a7200d-6acf-4008-8156-d9102730ac76-istio-podinfo\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q\" (UID: \"d8a7200d-6acf-4008-8156-d9102730ac76\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q"
Oct 30 19:29:41.701538 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.701516    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a7200d-6acf-4008-8156-d9102730ac76-workload-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q\" (UID: \"d8a7200d-6acf-4008-8156-d9102730ac76\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q"
Oct 30 19:29:41.701948 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.701572    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8a7200d-6acf-4008-8156-d9102730ac76-istio-envoy\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q\" (UID: \"d8a7200d-6acf-4008-8156-d9102730ac76\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q"
Oct 30 19:29:41.701948 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.701606    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8a7200d-6acf-4008-8156-d9102730ac76-workload-certs\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q\" (UID: \"d8a7200d-6acf-4008-8156-d9102730ac76\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q"
Oct 30 19:29:41.701948 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.701707    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8a7200d-6acf-4008-8156-d9102730ac76-istio-token\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q\" (UID: \"d8a7200d-6acf-4008-8156-d9102730ac76\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q"
Oct 30 19:29:41.701948 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.701754    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f43b74d6-8c5b-430d-bffb-a30beb6996ec-istiod-ca-cert\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb\" (UID: \"f43b74d6-8c5b-430d-bffb-a30beb6996ec\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb"
Oct 30 19:29:41.701948 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.701829    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f43b74d6-8c5b-430d-bffb-a30beb6996ec-istio-envoy\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb\" (UID: \"f43b74d6-8c5b-430d-bffb-a30beb6996ec\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb"
Oct 30 19:29:41.701948 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.701907    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43b74d6-8c5b-430d-bffb-a30beb6996ec-workload-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb\" (UID: \"f43b74d6-8c5b-430d-bffb-a30beb6996ec\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb"
Oct 30 19:29:41.702233 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.701952    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a7200d-6acf-4008-8156-d9102730ac76-credential-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q\" (UID: \"d8a7200d-6acf-4008-8156-d9102730ac76\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q"
Oct 30 19:29:41.702233 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.701979    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvb89\" (UniqueName: \"kubernetes.io/projected/f43b74d6-8c5b-430d-bffb-a30beb6996ec-kube-api-access-pvb89\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb\" (UID: \"f43b74d6-8c5b-430d-bffb-a30beb6996ec\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb"
Oct 30 19:29:41.702233 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.702011    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h9j8\" (UniqueName: \"kubernetes.io/projected/d8a7200d-6acf-4008-8156-d9102730ac76-kube-api-access-7h9j8\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q\" (UID: \"d8a7200d-6acf-4008-8156-d9102730ac76\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q"
Oct 30 19:29:41.702233 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.702072    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f43b74d6-8c5b-430d-bffb-a30beb6996ec-workload-certs\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb\" (UID: \"f43b74d6-8c5b-430d-bffb-a30beb6996ec\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb"
Oct 30 19:29:41.702233 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.702097    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f43b74d6-8c5b-430d-bffb-a30beb6996ec-istio-data\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb\" (UID: \"f43b74d6-8c5b-430d-bffb-a30beb6996ec\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb"
Oct 30 19:29:41.703706 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.703673    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8a7200d-6acf-4008-8156-d9102730ac76-istiod-ca-cert\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q\" (UID: \"d8a7200d-6acf-4008-8156-d9102730ac76\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q"
Oct 30 19:29:41.704062 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.704040    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8a7200d-6acf-4008-8156-d9102730ac76-istio-data\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q\" (UID: \"d8a7200d-6acf-4008-8156-d9102730ac76\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q"
Oct 30 19:29:41.704780 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.704756    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a7200d-6acf-4008-8156-d9102730ac76-credential-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q\" (UID: \"d8a7200d-6acf-4008-8156-d9102730ac76\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q"
Oct 30 19:29:41.705426 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.705397    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8a7200d-6acf-4008-8156-d9102730ac76-workload-certs\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q\" (UID: \"d8a7200d-6acf-4008-8156-d9102730ac76\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q"
Oct 30 19:29:41.705684 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.705660    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a7200d-6acf-4008-8156-d9102730ac76-workload-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q\" (UID: \"d8a7200d-6acf-4008-8156-d9102730ac76\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q"
Oct 30 19:29:41.727541 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.727509    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8a7200d-6acf-4008-8156-d9102730ac76-istio-podinfo\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q\" (UID: \"d8a7200d-6acf-4008-8156-d9102730ac76\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q"
Oct 30 19:29:41.730935 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.730905    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8a7200d-6acf-4008-8156-d9102730ac76-istio-envoy\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q\" (UID: \"d8a7200d-6acf-4008-8156-d9102730ac76\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q"
Oct 30 19:29:41.733850 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.733777    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8a7200d-6acf-4008-8156-d9102730ac76-istio-token\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q\" (UID: \"d8a7200d-6acf-4008-8156-d9102730ac76\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q"
Oct 30 19:29:41.742153 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.741940    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h9j8\" (UniqueName: \"kubernetes.io/projected/d8a7200d-6acf-4008-8156-d9102730ac76-kube-api-access-7h9j8\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q\" (UID: \"d8a7200d-6acf-4008-8156-d9102730ac76\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q"
Oct 30 19:29:41.806508 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.804873    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f43b74d6-8c5b-430d-bffb-a30beb6996ec-istio-envoy\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb\" (UID: \"f43b74d6-8c5b-430d-bffb-a30beb6996ec\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb"
Oct 30 19:29:41.806508 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.804960    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43b74d6-8c5b-430d-bffb-a30beb6996ec-workload-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb\" (UID: \"f43b74d6-8c5b-430d-bffb-a30beb6996ec\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb"
Oct 30 19:29:41.806508 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.805003    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvb89\" (UniqueName: \"kubernetes.io/projected/f43b74d6-8c5b-430d-bffb-a30beb6996ec-kube-api-access-pvb89\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb\" (UID: \"f43b74d6-8c5b-430d-bffb-a30beb6996ec\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb"
Oct 30 19:29:41.806508 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.805053    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f43b74d6-8c5b-430d-bffb-a30beb6996ec-workload-certs\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb\" (UID: \"f43b74d6-8c5b-430d-bffb-a30beb6996ec\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb"
Oct 30 19:29:41.806508 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.805082    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f43b74d6-8c5b-430d-bffb-a30beb6996ec-istio-data\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb\" (UID: \"f43b74d6-8c5b-430d-bffb-a30beb6996ec\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb"
Oct 30 19:29:41.806508 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.805206    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43b74d6-8c5b-430d-bffb-a30beb6996ec-credential-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb\" (UID: \"f43b74d6-8c5b-430d-bffb-a30beb6996ec\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb"
Oct 30 19:29:41.806508 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.805250    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f43b74d6-8c5b-430d-bffb-a30beb6996ec-istio-podinfo\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb\" (UID: \"f43b74d6-8c5b-430d-bffb-a30beb6996ec\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb"
Oct 30 19:29:41.806508 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.805279    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f43b74d6-8c5b-430d-bffb-a30beb6996ec-istio-token\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb\" (UID: \"f43b74d6-8c5b-430d-bffb-a30beb6996ec\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb"
Oct 30 19:29:41.806508 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.805440    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f43b74d6-8c5b-430d-bffb-a30beb6996ec-istiod-ca-cert\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb\" (UID: \"f43b74d6-8c5b-430d-bffb-a30beb6996ec\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb"
Oct 30 19:29:41.806508 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.806357    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f43b74d6-8c5b-430d-bffb-a30beb6996ec-istiod-ca-cert\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb\" (UID: \"f43b74d6-8c5b-430d-bffb-a30beb6996ec\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb"
Oct 30 19:29:41.807385 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.807328    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f43b74d6-8c5b-430d-bffb-a30beb6996ec-workload-certs\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb\" (UID: \"f43b74d6-8c5b-430d-bffb-a30beb6996ec\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb"
Oct 30 19:29:41.807385 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.807341    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f43b74d6-8c5b-430d-bffb-a30beb6996ec-istio-data\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb\" (UID: \"f43b74d6-8c5b-430d-bffb-a30beb6996ec\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb"
Oct 30 19:29:41.808003 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.807793    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43b74d6-8c5b-430d-bffb-a30beb6996ec-workload-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb\" (UID: \"f43b74d6-8c5b-430d-bffb-a30beb6996ec\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb"
Oct 30 19:29:41.808111 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.808060    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43b74d6-8c5b-430d-bffb-a30beb6996ec-credential-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb\" (UID: \"f43b74d6-8c5b-430d-bffb-a30beb6996ec\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb"
Oct 30 19:29:41.809408 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.809356    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f43b74d6-8c5b-430d-bffb-a30beb6996ec-istio-envoy\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb\" (UID: \"f43b74d6-8c5b-430d-bffb-a30beb6996ec\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb"
Oct 30 19:29:41.810516 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.810466    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f43b74d6-8c5b-430d-bffb-a30beb6996ec-istio-podinfo\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb\" (UID: \"f43b74d6-8c5b-430d-bffb-a30beb6996ec\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb"
Oct 30 19:29:41.825511 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.825427    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvb89\" (UniqueName: \"kubernetes.io/projected/f43b74d6-8c5b-430d-bffb-a30beb6996ec-kube-api-access-pvb89\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb\" (UID: \"f43b74d6-8c5b-430d-bffb-a30beb6996ec\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb"
Oct 30 19:29:41.826242 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.826211    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f43b74d6-8c5b-430d-bffb-a30beb6996ec-istio-token\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb\" (UID: \"f43b74d6-8c5b-430d-bffb-a30beb6996ec\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb"
Oct 30 19:29:41.917104 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.917066    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q"
Oct 30 19:29:41.954566 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.954202    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-799546ec-lnflr"
Oct 30 19:29:41.992809 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.992190    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb"
Oct 30 19:29:41.995379 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:41.995345    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2b9fef5-vfw6s"]
Oct 30 19:29:42.001720 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:29:42.001674    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad59b8d8_2911_4fe8_b7cd_9090ff58590c.slice/crio-4e99407d05aa0565f318b04bf66e0bf210f67e7c47c73811619870c3dc84c160 WatchSource:0}: Error finding container 4e99407d05aa0565f318b04bf66e0bf210f67e7c47c73811619870c3dc84c160: Status 404 returned error can't find the container with id 4e99407d05aa0565f318b04bf66e0bf210f67e7c47c73811619870c3dc84c160
Oct 30 19:29:42.009770 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.007633    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51267af2-5e1b-438e-b1be-25f8f5ee2d9f-kube-api-access-z2xpz\" (UniqueName: \"kubernetes.io/projected/51267af2-5e1b-438e-b1be-25f8f5ee2d9f-kube-api-access-z2xpz\") pod \"51267af2-5e1b-438e-b1be-25f8f5ee2d9f\" (UID: \"51267af2-5e1b-438e-b1be-25f8f5ee2d9f\") "
Oct 30 19:29:42.009770 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.007929    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51267af2-5e1b-438e-b1be-25f8f5ee2d9f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51267af2-5e1b-438e-b1be-25f8f5ee2d9f-maas-api-openshift-ca\") pod \"51267af2-5e1b-438e-b1be-25f8f5ee2d9f\" (UID: \"51267af2-5e1b-438e-b1be-25f8f5ee2d9f\") "
Oct 30 19:29:42.009770 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.007979    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51267af2-5e1b-438e-b1be-25f8f5ee2d9f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51267af2-5e1b-438e-b1be-25f8f5ee2d9f-maas-api-xks-ca\") pod \"51267af2-5e1b-438e-b1be-25f8f5ee2d9f\" (UID: \"51267af2-5e1b-438e-b1be-25f8f5ee2d9f\") "
Oct 30 19:29:42.011617 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.011273    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51267af2-5e1b-438e-b1be-25f8f5ee2d9f-maas-api-openshift-ca" pod "51267af2-5e1b-438e-b1be-25f8f5ee2d9f" (UID: "51267af2-5e1b-438e-b1be-25f8f5ee2d9f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:29:42.014276 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.014201    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51267af2-5e1b-438e-b1be-25f8f5ee2d9f-kube-api-access-z2xpz" pod "51267af2-5e1b-438e-b1be-25f8f5ee2d9f" (UID: "51267af2-5e1b-438e-b1be-25f8f5ee2d9f"). InnerVolumeSpecName "kube-api-access-z2xpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:29:42.021186 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.021111    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51267af2-5e1b-438e-b1be-25f8f5ee2d9f-maas-api-xks-ca" pod "51267af2-5e1b-438e-b1be-25f8f5ee2d9f" (UID: "51267af2-5e1b-438e-b1be-25f8f5ee2d9f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:29:42.110479 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.110439    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2xpz\" (UniqueName: \"kubernetes.io/projected/51267af2-5e1b-438e-b1be-25f8f5ee2d9f-kube-api-access-z2xpz\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:42.110479 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.110480    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51267af2-5e1b-438e-b1be-25f8f5ee2d9f-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:42.110761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.110498    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51267af2-5e1b-438e-b1be-25f8f5ee2d9f-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:42.185433 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.185362    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv"]
Oct 30 19:29:42.190208 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.190167    5043 cpu_manager.go:414] "containerMap: removing container" podUID="51267af2-5e1b-438e-b1be-25f8f5ee2d9f" containerName="revoke-keys"
Oct 30 19:29:42.190208 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.190196    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51267af2-5e1b-438e-b1be-25f8f5ee2d9f" containerName="revoke-keys"
Oct 30 19:29:42.190508 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.190426    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="51267af2-5e1b-438e-b1be-25f8f5ee2d9f" containerName="revoke-keys"
Oct 30 19:29:42.199438 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.199408    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv"
Oct 30 19:29:42.203661 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.203635    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv"]
Oct 30 19:29:42.215597 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.214905    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj9xg\" (UniqueName: \"kubernetes.io/projected/23f8f6fc-85c3-46ac-88f5-08e08f246560-kube-api-access-zj9xg\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv\" (UID: \"23f8f6fc-85c3-46ac-88f5-08e08f246560\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv"
Oct 30 19:29:42.215597 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.214956    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f8f6fc-85c3-46ac-88f5-08e08f246560-credential-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv\" (UID: \"23f8f6fc-85c3-46ac-88f5-08e08f246560\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv"
Oct 30 19:29:42.215597 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.214982    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f8f6fc-85c3-46ac-88f5-08e08f246560-istio-data\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv\" (UID: \"23f8f6fc-85c3-46ac-88f5-08e08f246560\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv"
Oct 30 19:29:42.215597 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.215062    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f8f6fc-85c3-46ac-88f5-08e08f246560-istiod-ca-cert\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv\" (UID: \"23f8f6fc-85c3-46ac-88f5-08e08f246560\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv"
Oct 30 19:29:42.215597 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.215214    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f8f6fc-85c3-46ac-88f5-08e08f246560-workload-certs\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv\" (UID: \"23f8f6fc-85c3-46ac-88f5-08e08f246560\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv"
Oct 30 19:29:42.215597 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.215258    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f8f6fc-85c3-46ac-88f5-08e08f246560-istio-envoy\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv\" (UID: \"23f8f6fc-85c3-46ac-88f5-08e08f246560\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv"
Oct 30 19:29:42.215597 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.215307    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f8f6fc-85c3-46ac-88f5-08e08f246560-istio-podinfo\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv\" (UID: \"23f8f6fc-85c3-46ac-88f5-08e08f246560\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv"
Oct 30 19:29:42.215597 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.215380    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f8f6fc-85c3-46ac-88f5-08e08f246560-workload-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv\" (UID: \"23f8f6fc-85c3-46ac-88f5-08e08f246560\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv"
Oct 30 19:29:42.216204 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.215645    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23f8f6fc-85c3-46ac-88f5-08e08f246560-istio-token\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv\" (UID: \"23f8f6fc-85c3-46ac-88f5-08e08f246560\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv"
Oct 30 19:29:42.256698 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.256633    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q"]
Oct 30 19:29:42.260513 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:29:42.260479    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8a7200d_6acf_4008_8156_d9102730ac76.slice/crio-a8e2500cf7b15c833686ab1c4bcd6f952d5d308771e5abc4986413fb6253fbdf WatchSource:0}: Error finding container a8e2500cf7b15c833686ab1c4bcd6f952d5d308771e5abc4986413fb6253fbdf: Status 404 returned error can't find the container with id a8e2500cf7b15c833686ab1c4bcd6f952d5d308771e5abc4986413fb6253fbdf
Oct 30 19:29:42.263959 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.263922    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:29:42.264044 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.264013    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:29:42.264081 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.264055    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:29:42.317060 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.317036    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f8f6fc-85c3-46ac-88f5-08e08f246560-istio-podinfo\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv\" (UID: \"23f8f6fc-85c3-46ac-88f5-08e08f246560\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv"
Oct 30 19:29:42.317138 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.317069    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f8f6fc-85c3-46ac-88f5-08e08f246560-workload-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv\" (UID: \"23f8f6fc-85c3-46ac-88f5-08e08f246560\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv"
Oct 30 19:29:42.317180 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.317134    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23f8f6fc-85c3-46ac-88f5-08e08f246560-istio-token\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv\" (UID: \"23f8f6fc-85c3-46ac-88f5-08e08f246560\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv"
Oct 30 19:29:42.318928 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.317204    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj9xg\" (UniqueName: \"kubernetes.io/projected/23f8f6fc-85c3-46ac-88f5-08e08f246560-kube-api-access-zj9xg\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv\" (UID: \"23f8f6fc-85c3-46ac-88f5-08e08f246560\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv"
Oct 30 19:29:42.318928 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.317235    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f8f6fc-85c3-46ac-88f5-08e08f246560-credential-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv\" (UID: \"23f8f6fc-85c3-46ac-88f5-08e08f246560\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv"
Oct 30 19:29:42.318928 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.317301    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f8f6fc-85c3-46ac-88f5-08e08f246560-istio-data\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv\" (UID: \"23f8f6fc-85c3-46ac-88f5-08e08f246560\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv"
Oct 30 19:29:42.318928 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.317398    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f8f6fc-85c3-46ac-88f5-08e08f246560-istiod-ca-cert\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv\" (UID: \"23f8f6fc-85c3-46ac-88f5-08e08f246560\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv"
Oct 30 19:29:42.318928 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.317616    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f8f6fc-85c3-46ac-88f5-08e08f246560-workload-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv\" (UID: \"23f8f6fc-85c3-46ac-88f5-08e08f246560\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv"
Oct 30 19:29:42.318928 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.317768    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f8f6fc-85c3-46ac-88f5-08e08f246560-workload-certs\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv\" (UID: \"23f8f6fc-85c3-46ac-88f5-08e08f246560\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv"
Oct 30 19:29:42.318928 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.317833    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f8f6fc-85c3-46ac-88f5-08e08f246560-istio-envoy\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv\" (UID: \"23f8f6fc-85c3-46ac-88f5-08e08f246560\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv"
Oct 30 19:29:42.318928 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.317929    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f8f6fc-85c3-46ac-88f5-08e08f246560-credential-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv\" (UID: \"23f8f6fc-85c3-46ac-88f5-08e08f246560\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv"
Oct 30 19:29:42.318928 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.318057    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f8f6fc-85c3-46ac-88f5-08e08f246560-istio-data\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv\" (UID: \"23f8f6fc-85c3-46ac-88f5-08e08f246560\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv"
Oct 30 19:29:42.318928 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.318121    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f8f6fc-85c3-46ac-88f5-08e08f246560-workload-certs\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv\" (UID: \"23f8f6fc-85c3-46ac-88f5-08e08f246560\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv"
Oct 30 19:29:42.318928 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.318275    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f8f6fc-85c3-46ac-88f5-08e08f246560-istiod-ca-cert\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv\" (UID: \"23f8f6fc-85c3-46ac-88f5-08e08f246560\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv"
Oct 30 19:29:42.320959 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.320864    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f8f6fc-85c3-46ac-88f5-08e08f246560-istio-podinfo\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv\" (UID: \"23f8f6fc-85c3-46ac-88f5-08e08f246560\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv"
Oct 30 19:29:42.322831 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.322803    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb"]
Oct 30 19:29:42.326131 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.326100    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f8f6fc-85c3-46ac-88f5-08e08f246560-istio-envoy\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv\" (UID: \"23f8f6fc-85c3-46ac-88f5-08e08f246560\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv"
Oct 30 19:29:42.328707 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.328685    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23f8f6fc-85c3-46ac-88f5-08e08f246560-istio-token\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv\" (UID: \"23f8f6fc-85c3-46ac-88f5-08e08f246560\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv"
Oct 30 19:29:42.329710 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.329669    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:29:42.329851 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.329823    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:29:42.329928 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.329907    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:29:42.330519 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.330499    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj9xg\" (UniqueName: \"kubernetes.io/projected/23f8f6fc-85c3-46ac-88f5-08e08f246560-kube-api-access-zj9xg\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv\" (UID: \"23f8f6fc-85c3-46ac-88f5-08e08f246560\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv"
Oct 30 19:29:42.525178 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.525152    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv"
Oct 30 19:29:42.745458 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.745346    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b9fef5-vfw6s" event={"ID":"ad59b8d8-2911-4fe8-b7cd-9090ff58590c","Type":"ContainerStarted","Data":"f4d18cce38c8e51725329d076658f012568c5b4a2da12ea6823c383618c9aa64"}
Oct 30 19:29:42.745458 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.745421    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b9fef5-vfw6s" event={"ID":"ad59b8d8-2911-4fe8-b7cd-9090ff58590c","Type":"ContainerStarted","Data":"4e99407d05aa0565f318b04bf66e0bf210f67e7c47c73811619870c3dc84c160"}
Oct 30 19:29:42.748390 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.747594    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-799546ec-lnflr" event={"ID":"51267af2-5e1b-438e-b1be-25f8f5ee2d9f","Type":"ContainerDied","Data":"3a8f687eabab5a5e36e36f2954069a139f65ef9e9ba224ded6aa85a5222c7685"}
Oct 30 19:29:42.748390 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.747643    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a8f687eabab5a5e36e36f2954069a139f65ef9e9ba224ded6aa85a5222c7685"
Oct 30 19:29:42.748390 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.747728    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-799546ec-lnflr"
Oct 30 19:29:42.753598 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.752842    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q" event={"ID":"d8a7200d-6acf-4008-8156-d9102730ac76","Type":"ContainerStarted","Data":"200b1ca9a3dfa73fa6d763f3b5eefb5d256b363ec4d4ae282a751afd3c04377d"}
Oct 30 19:29:42.753598 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.752895    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q" event={"ID":"d8a7200d-6acf-4008-8156-d9102730ac76","Type":"ContainerStarted","Data":"a8e2500cf7b15c833686ab1c4bcd6f952d5d308771e5abc4986413fb6253fbdf"}
Oct 30 19:29:42.823299 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.817560    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb" event={"ID":"f43b74d6-8c5b-430d-bffb-a30beb6996ec","Type":"ContainerStarted","Data":"2ba4d9175cbdb92892dec3a9fe3684e7413884e3fd7380ded51b886fef7fe7ec"}
Oct 30 19:29:42.823299 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.817635    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb" event={"ID":"f43b74d6-8c5b-430d-bffb-a30beb6996ec","Type":"ContainerStarted","Data":"cab3a751b89d9ffd77985f0511bb7f300c6b018e3096a3e4c8820ba171a4a268"}
Oct 30 19:29:42.823299 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.819390    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b9fef5-vfw6s" podStartSLOduration=1.819372005 podStartE2EDuration="1.819372005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:42.816431029 +0000 UTC m=+802.327490985" watchObservedRunningTime="2026-10-30 19:29:42.819372005 +0000 UTC m=+802.330431988"
Oct 30 19:29:42.864769 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.863639    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q" podStartSLOduration=1.8636203199999999 podStartE2EDuration="1.86362032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:42.861397829 +0000 UTC m=+802.372457777" watchObservedRunningTime="2026-10-30 19:29:42.86362032 +0000 UTC m=+802.374680286"
Oct 30 19:29:42.875148 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.875104    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:29:42.875283 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.875197    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:29:42.875283 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.875240    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:29:42.882012 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.879587    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv"]
Oct 30 19:29:42.898905 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.898843    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb" podStartSLOduration=1.898820594 podStartE2EDuration="1.898820594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:42.890282122 +0000 UTC m=+802.401342088" watchObservedRunningTime="2026-10-30 19:29:42.898820594 +0000 UTC m=+802.409880564"
Oct 30 19:29:42.920083 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.920044    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q"
Oct 30 19:29:43.001766 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:42.998117    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb"
Oct 30 19:29:43.857787 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:43.856634    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv" event={"ID":"23f8f6fc-85c3-46ac-88f5-08e08f246560","Type":"ContainerStarted","Data":"8b068dfe8707cc390ec583f8fa4bc639edaf1fdabf87227a1d758ab4b49d149e"}
Oct 30 19:29:43.857787 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:43.856706    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv" event={"ID":"23f8f6fc-85c3-46ac-88f5-08e08f246560","Type":"ContainerStarted","Data":"c93938c7040ac388e19101d773e024ffb103524bff47e765f9503fe4047442d5"}
Oct 30 19:29:43.907316 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:43.907280    5043 generic.go:356] "Generic (PLEG): container finished" podID="ad59b8d8-2911-4fe8-b7cd-9090ff58590c" containerID="f4d18cce38c8e51725329d076658f012568c5b4a2da12ea6823c383618c9aa64" exitCode=0
Oct 30 19:29:43.918758 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:43.908606    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b9fef5-vfw6s" event={"ID":"ad59b8d8-2911-4fe8-b7cd-9090ff58590c","Type":"ContainerDied","Data":"f4d18cce38c8e51725329d076658f012568c5b4a2da12ea6823c383618c9aa64"}
Oct 30 19:29:43.922863 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:43.922817    5043 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 19:29:43.923114 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:43.923080    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q" podUID="d8a7200d-6acf-4008-8156-d9102730ac76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 19:29:43.957725 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:43.957655    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv" podStartSLOduration=1.957631914 podStartE2EDuration="1.957631914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:43.924004324 +0000 UTC m=+803.435064287" watchObservedRunningTime="2026-10-30 19:29:43.957631914 +0000 UTC m=+803.468691882"
Oct 30 19:29:43.994412 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:43.994376    5043 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 19:29:43.994661 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:43.994637    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb" podUID="f43b74d6-8c5b-430d-bffb-a30beb6996ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 19:29:44.529799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:44.529763    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv"
Oct 30 19:29:44.535768 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:44.532848    5043 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 19:29:44.535768 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:44.532910    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv" podUID="23f8f6fc-85c3-46ac-88f5-08e08f246560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 19:29:44.925958 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:44.925857    5043 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 19:29:44.926595 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:44.926559    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q" podUID="d8a7200d-6acf-4008-8156-d9102730ac76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 19:29:44.994826 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:44.992589    5043 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 19:29:44.994826 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:44.992656    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb" podUID="f43b74d6-8c5b-430d-bffb-a30beb6996ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 19:29:45.631822 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:45.615181    5043 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 19:29:45.631822 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:45.615244    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv" podUID="23f8f6fc-85c3-46ac-88f5-08e08f246560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 19:29:45.678794 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:45.677311    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:29:45.940172 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:45.935702    5043 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 19:29:45.940172 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:45.935780    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q" podUID="d8a7200d-6acf-4008-8156-d9102730ac76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 19:29:46.002165 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:46.001689    5043 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 19:29:46.002165 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:46.001771    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb" podUID="f43b74d6-8c5b-430d-bffb-a30beb6996ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 19:29:46.151764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:46.137442    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b9fef5-vfw6s"
Oct 30 19:29:46.198359 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:46.197451    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:46.433762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:46.419781    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad59b8d8-2911-4fe8-b7cd-9090ff58590c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad59b8d8-2911-4fe8-b7cd-9090ff58590c-maas-api-openshift-ca\") pod \"ad59b8d8-2911-4fe8-b7cd-9090ff58590c\" (UID: \"ad59b8d8-2911-4fe8-b7cd-9090ff58590c\") "
Oct 30 19:29:46.433762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:46.419898    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad59b8d8-2911-4fe8-b7cd-9090ff58590c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad59b8d8-2911-4fe8-b7cd-9090ff58590c-maas-api-xks-ca\") pod \"ad59b8d8-2911-4fe8-b7cd-9090ff58590c\" (UID: \"ad59b8d8-2911-4fe8-b7cd-9090ff58590c\") "
Oct 30 19:29:46.433762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:46.420048    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad59b8d8-2911-4fe8-b7cd-9090ff58590c-kube-api-access-ss8hs\" (UniqueName: \"kubernetes.io/projected/ad59b8d8-2911-4fe8-b7cd-9090ff58590c-kube-api-access-ss8hs\") pod \"ad59b8d8-2911-4fe8-b7cd-9090ff58590c\" (UID: \"ad59b8d8-2911-4fe8-b7cd-9090ff58590c\") "
Oct 30 19:29:46.433762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:46.430935    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad59b8d8-2911-4fe8-b7cd-9090ff58590c-maas-api-openshift-ca" pod "ad59b8d8-2911-4fe8-b7cd-9090ff58590c" (UID: "ad59b8d8-2911-4fe8-b7cd-9090ff58590c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:29:46.488764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:46.476898    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad59b8d8-2911-4fe8-b7cd-9090ff58590c-maas-api-xks-ca" pod "ad59b8d8-2911-4fe8-b7cd-9090ff58590c" (UID: "ad59b8d8-2911-4fe8-b7cd-9090ff58590c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:29:46.492761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:46.491921    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad59b8d8-2911-4fe8-b7cd-9090ff58590c-kube-api-access-ss8hs" pod "ad59b8d8-2911-4fe8-b7cd-9090ff58590c" (UID: "ad59b8d8-2911-4fe8-b7cd-9090ff58590c"). InnerVolumeSpecName "kube-api-access-ss8hs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:29:46.529995 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:46.526247    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad59b8d8-2911-4fe8-b7cd-9090ff58590c-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:46.529995 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:46.526290    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad59b8d8-2911-4fe8-b7cd-9090ff58590c-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:46.529995 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:46.526310    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss8hs\" (UniqueName: \"kubernetes.io/projected/ad59b8d8-2911-4fe8-b7cd-9090ff58590c-kube-api-access-ss8hs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:46.529995 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:46.527539    5043 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 19:29:46.529995 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:46.527605    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv" podUID="23f8f6fc-85c3-46ac-88f5-08e08f246560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 19:29:46.625760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:46.625422    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded"
Oct 30 19:29:46.719841 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:46.698234    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:46.719841 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:46.718199    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:46.921258 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:46.921202    5043 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 19:29:46.921635 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:46.921581    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q" podUID="d8a7200d-6acf-4008-8156-d9102730ac76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 19:29:46.992505 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:46.992472    5043 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 19:29:46.993353 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:46.993237    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb" podUID="f43b74d6-8c5b-430d-bffb-a30beb6996ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 19:29:47.124818 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:47.124389    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b9fef5-vfw6s" event={"ID":"ad59b8d8-2911-4fe8-b7cd-9090ff58590c","Type":"ContainerDied","Data":"4e99407d05aa0565f318b04bf66e0bf210f67e7c47c73811619870c3dc84c160"}
Oct 30 19:29:47.124818 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:47.124456    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e99407d05aa0565f318b04bf66e0bf210f67e7c47c73811619870c3dc84c160"
Oct 30 19:29:47.124818 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:47.124526    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b9fef5-vfw6s"
Oct 30 19:29:47.209771 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:47.207836    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:47.209771 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:47.208653    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:29:47.209771 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:47.209567    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:47.220846 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:47.219788    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:47.318767 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:47.309049    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:47.539763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:47.536194    5043 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 19:29:47.539763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:47.536257    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv" podUID="23f8f6fc-85c3-46ac-88f5-08e08f246560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 19:29:47.685953 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:47.685896    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:47.924820 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:47.924643    5043 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 19:29:47.924820 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:47.924712    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q" podUID="d8a7200d-6acf-4008-8156-d9102730ac76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 19:29:47.953644 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:47.953612    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:29:47.996760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:47.993173    5043 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 19:29:47.996760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:47.993232    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb" podUID="f43b74d6-8c5b-430d-bffb-a30beb6996ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 19:29:48.193821 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:48.185874    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:48.342874 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:48.342829    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:48.365764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:48.361253    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:29:48.478814 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:48.476839    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:48.638768 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:48.615945    5043 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 19:29:48.638768 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:48.616024    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv" podUID="23f8f6fc-85c3-46ac-88f5-08e08f246560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 19:29:48.731814 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:48.731353    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:48.922270 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:48.922229    5043 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 19:29:48.922576 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:48.922529    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q" podUID="d8a7200d-6acf-4008-8156-d9102730ac76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 19:29:48.959773 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:48.957325    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:49.000821 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:49.000426    5043 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 19:29:49.000821 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:49.000507    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb" podUID="f43b74d6-8c5b-430d-bffb-a30beb6996ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 19:29:49.154784 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:49.154754    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:29:49.526461 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:49.526428    5043 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 19:29:49.526712 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:49.526690    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv" podUID="23f8f6fc-85c3-46ac-88f5-08e08f246560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 19:29:49.920868 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:49.920778    5043 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Oct 30 19:29:49.920868 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:49.920833    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q" podUID="d8a7200d-6acf-4008-8156-d9102730ac76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Oct 30 19:29:49.993366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:49.993316    5043 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Oct 30 19:29:49.993574 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:49.993392    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb" podUID="f43b74d6-8c5b-430d-bffb-a30beb6996ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Oct 30 19:29:50.527142 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:50.527100    5043 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 30 19:29:50.527682 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:50.527172    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv" podUID="23f8f6fc-85c3-46ac-88f5-08e08f246560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 30 19:29:51.078147 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:51.078115    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q"
Oct 30 19:29:51.081505 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:51.081480    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q"
Oct 30 19:29:51.081960 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:51.081943    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q"
Oct 30 19:29:51.216817 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:51.216784    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb"
Oct 30 19:29:51.218913 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:51.218889    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb"
Oct 30 19:29:51.220522 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:51.220498    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb"
Oct 30 19:29:51.660521 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:51.660465    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8966f23c-llfvq"]
Oct 30 19:29:51.661501 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:51.661474    5043 cpu_manager.go:414] "containerMap: removing container" podUID="ad59b8d8-2911-4fe8-b7cd-9090ff58590c" containerName="revoke-keys"
Oct 30 19:29:51.661691 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:51.661650    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad59b8d8-2911-4fe8-b7cd-9090ff58590c" containerName="revoke-keys"
Oct 30 19:29:51.661952 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:51.661935    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad59b8d8-2911-4fe8-b7cd-9090ff58590c" containerName="revoke-keys"
Oct 30 19:29:51.726891 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:51.726857    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8966f23c-llfvq"]
Oct 30 19:29:51.732903 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:51.727082    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8966f23c-llfvq"
Oct 30 19:29:51.737638 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:51.737562    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:29:51.738079 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:51.738039    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:29:51.738341 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:51.738310    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:29:51.738582 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:51.738539    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:29:51.812364 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:51.812043    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a8a9a63-wh84h"]
Oct 30 19:29:51.818817 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:51.818789    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a8a9a63-wh84h"
Oct 30 19:29:51.822893 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:51.822621    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a8a9a63-wh84h"]
Oct 30 19:29:51.879947 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:51.879915    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv"
Oct 30 19:29:51.880790 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:51.880702    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv"
Oct 30 19:29:51.881987 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:51.881965    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv"
Oct 30 19:29:51.910434 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:51.910395    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kdgw\" (UniqueName: \"kubernetes.io/projected/fb02228c-861e-41bb-b16e-31eb93072532-kube-api-access-2kdgw\") pod \"maas-api-revoke-sub-8966f23c-llfvq\" (UID: \"fb02228c-861e-41bb-b16e-31eb93072532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8966f23c-llfvq"
Oct 30 19:29:51.910640 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:51.910570    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkzt4\" (UniqueName: \"kubernetes.io/projected/52b13089-acb8-4eee-a734-04113162fca4-kube-api-access-qkzt4\") pod \"maas-api-revoke-sub-9a8a9a63-wh84h\" (UID: \"52b13089-acb8-4eee-a734-04113162fca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a8a9a63-wh84h"
Oct 30 19:29:51.910867 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:51.910786    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb02228c-861e-41bb-b16e-31eb93072532-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8966f23c-llfvq\" (UID: \"fb02228c-861e-41bb-b16e-31eb93072532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8966f23c-llfvq"
Oct 30 19:29:51.911016 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:51.910969    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb02228c-861e-41bb-b16e-31eb93072532-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8966f23c-llfvq\" (UID: \"fb02228c-861e-41bb-b16e-31eb93072532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8966f23c-llfvq"
Oct 30 19:29:51.911080 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:51.911040    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52b13089-acb8-4eee-a734-04113162fca4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a8a9a63-wh84h\" (UID: \"52b13089-acb8-4eee-a734-04113162fca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a8a9a63-wh84h"
Oct 30 19:29:51.911137 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:51.911085    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52b13089-acb8-4eee-a734-04113162fca4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a8a9a63-wh84h\" (UID: \"52b13089-acb8-4eee-a734-04113162fca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a8a9a63-wh84h"
Oct 30 19:29:52.012122 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:52.012090    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb02228c-861e-41bb-b16e-31eb93072532-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8966f23c-llfvq\" (UID: \"fb02228c-861e-41bb-b16e-31eb93072532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8966f23c-llfvq"
Oct 30 19:29:52.012313 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:52.012160    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52b13089-acb8-4eee-a734-04113162fca4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a8a9a63-wh84h\" (UID: \"52b13089-acb8-4eee-a734-04113162fca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a8a9a63-wh84h"
Oct 30 19:29:52.012313 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:52.012196    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52b13089-acb8-4eee-a734-04113162fca4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a8a9a63-wh84h\" (UID: \"52b13089-acb8-4eee-a734-04113162fca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a8a9a63-wh84h"
Oct 30 19:29:52.012313 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:52.012278    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kdgw\" (UniqueName: \"kubernetes.io/projected/fb02228c-861e-41bb-b16e-31eb93072532-kube-api-access-2kdgw\") pod \"maas-api-revoke-sub-8966f23c-llfvq\" (UID: \"fb02228c-861e-41bb-b16e-31eb93072532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8966f23c-llfvq"
Oct 30 19:29:52.012486 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:52.012380    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkzt4\" (UniqueName: \"kubernetes.io/projected/52b13089-acb8-4eee-a734-04113162fca4-kube-api-access-qkzt4\") pod \"maas-api-revoke-sub-9a8a9a63-wh84h\" (UID: \"52b13089-acb8-4eee-a734-04113162fca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a8a9a63-wh84h"
Oct 30 19:29:52.012540 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:52.012498    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb02228c-861e-41bb-b16e-31eb93072532-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8966f23c-llfvq\" (UID: \"fb02228c-861e-41bb-b16e-31eb93072532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8966f23c-llfvq"
Oct 30 19:29:52.013348 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:52.013322    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb02228c-861e-41bb-b16e-31eb93072532-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8966f23c-llfvq\" (UID: \"fb02228c-861e-41bb-b16e-31eb93072532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8966f23c-llfvq"
Oct 30 19:29:52.014257 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:52.014232    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52b13089-acb8-4eee-a734-04113162fca4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a8a9a63-wh84h\" (UID: \"52b13089-acb8-4eee-a734-04113162fca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a8a9a63-wh84h"
Oct 30 19:29:52.015070 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:52.015044    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52b13089-acb8-4eee-a734-04113162fca4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a8a9a63-wh84h\" (UID: \"52b13089-acb8-4eee-a734-04113162fca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a8a9a63-wh84h"
Oct 30 19:29:52.015241 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:52.015205    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb02228c-861e-41bb-b16e-31eb93072532-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8966f23c-llfvq\" (UID: \"fb02228c-861e-41bb-b16e-31eb93072532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8966f23c-llfvq"
Oct 30 19:29:52.021424 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:52.021396    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkzt4\" (UniqueName: \"kubernetes.io/projected/52b13089-acb8-4eee-a734-04113162fca4-kube-api-access-qkzt4\") pod \"maas-api-revoke-sub-9a8a9a63-wh84h\" (UID: \"52b13089-acb8-4eee-a734-04113162fca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a8a9a63-wh84h"
Oct 30 19:29:52.021523 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:52.021420    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kdgw\" (UniqueName: \"kubernetes.io/projected/fb02228c-861e-41bb-b16e-31eb93072532-kube-api-access-2kdgw\") pod \"maas-api-revoke-sub-8966f23c-llfvq\" (UID: \"fb02228c-861e-41bb-b16e-31eb93072532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8966f23c-llfvq"
Oct 30 19:29:52.077333 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:52.077302    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8966f23c-llfvq"
Oct 30 19:29:52.136453 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:52.136014    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a8a9a63-wh84h"
Oct 30 19:29:52.368318 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:29:52.368274    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb02228c_861e_41bb_b16e_31eb93072532.slice/crio-c226aec534e287807cacc71647828c92fd5f0a2ed2bb3788a77fefa686fce93e WatchSource:0}: Error finding container c226aec534e287807cacc71647828c92fd5f0a2ed2bb3788a77fefa686fce93e: Status 404 returned error can't find the container with id c226aec534e287807cacc71647828c92fd5f0a2ed2bb3788a77fefa686fce93e
Oct 30 19:29:52.371166 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:52.371111    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8966f23c-llfvq"]
Oct 30 19:29:52.505168 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:52.504997    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a8a9a63-wh84h"]
Oct 30 19:29:52.509895 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:52.509855    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8966f23c-llfvq" event={"ID":"fb02228c-861e-41bb-b16e-31eb93072532","Type":"ContainerStarted","Data":"c226aec534e287807cacc71647828c92fd5f0a2ed2bb3788a77fefa686fce93e"}
Oct 30 19:29:52.559252 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:29:52.559213    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52b13089_acb8_4eee_a734_04113162fca4.slice/crio-43110050a10f0603e889ded23aaddc3dcbb0cf5152765e0c530759398328554b WatchSource:0}: Error finding container 43110050a10f0603e889ded23aaddc3dcbb0cf5152765e0c530759398328554b: Status 404 returned error can't find the container with id 43110050a10f0603e889ded23aaddc3dcbb0cf5152765e0c530759398328554b
Oct 30 19:29:53.528132 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:53.528096    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a8a9a63-wh84h" event={"ID":"52b13089-acb8-4eee-a734-04113162fca4","Type":"ContainerStarted","Data":"50ca142dc7057ad4e9c2e502e02d8f7ab60638ef8ae5fefb3f13c9073fb60278"}
Oct 30 19:29:53.528625 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:53.528159    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a8a9a63-wh84h" event={"ID":"52b13089-acb8-4eee-a734-04113162fca4","Type":"ContainerStarted","Data":"43110050a10f0603e889ded23aaddc3dcbb0cf5152765e0c530759398328554b"}
Oct 30 19:29:53.529913 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:53.529888    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8966f23c-llfvq" event={"ID":"fb02228c-861e-41bb-b16e-31eb93072532","Type":"ContainerStarted","Data":"85ed113461217a29b2913b30b659dd369fe6d289b2b04b938d3215a61cde4466"}
Oct 30 19:29:53.550318 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:53.550274    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a8a9a63-wh84h" podStartSLOduration=2.5502606009999997 podStartE2EDuration="2.550260601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:53.543676119 +0000 UTC m=+813.054736084" watchObservedRunningTime="2026-10-30 19:29:53.550260601 +0000 UTC m=+813.061320565"
Oct 30 19:29:53.565820 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:53.565770    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8966f23c-llfvq" podStartSLOduration=2.565753123 podStartE2EDuration="2.565753123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:53.559698995 +0000 UTC m=+813.070758961" watchObservedRunningTime="2026-10-30 19:29:53.565753123 +0000 UTC m=+813.076813085"
Oct 30 19:29:53.841515 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:53.841429    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d797f569-bfs6w"]
Oct 30 19:29:53.845995 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:53.845971    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d797f569-bfs6w"
Oct 30 19:29:53.856311 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:53.855683    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d797f569-bfs6w"]
Oct 30 19:29:53.945691 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:53.945654    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ktzv\" (UniqueName: \"kubernetes.io/projected/f911451a-2556-49ed-9778-7faa6549d341-kube-api-access-9ktzv\") pod \"maas-api-revoke-sub-d797f569-bfs6w\" (UID: \"f911451a-2556-49ed-9778-7faa6549d341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d797f569-bfs6w"
Oct 30 19:29:53.945902 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:53.945721    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f911451a-2556-49ed-9778-7faa6549d341-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d797f569-bfs6w\" (UID: \"f911451a-2556-49ed-9778-7faa6549d341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d797f569-bfs6w"
Oct 30 19:29:53.945902 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:53.945889    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f911451a-2556-49ed-9778-7faa6549d341-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d797f569-bfs6w\" (UID: \"f911451a-2556-49ed-9778-7faa6549d341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d797f569-bfs6w"
Oct 30 19:29:54.047233 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:54.047200    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ktzv\" (UniqueName: \"kubernetes.io/projected/f911451a-2556-49ed-9778-7faa6549d341-kube-api-access-9ktzv\") pod \"maas-api-revoke-sub-d797f569-bfs6w\" (UID: \"f911451a-2556-49ed-9778-7faa6549d341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d797f569-bfs6w"
Oct 30 19:29:54.047401 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:54.047280    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f911451a-2556-49ed-9778-7faa6549d341-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d797f569-bfs6w\" (UID: \"f911451a-2556-49ed-9778-7faa6549d341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d797f569-bfs6w"
Oct 30 19:29:54.047401 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:54.047389    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f911451a-2556-49ed-9778-7faa6549d341-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d797f569-bfs6w\" (UID: \"f911451a-2556-49ed-9778-7faa6549d341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d797f569-bfs6w"
Oct 30 19:29:54.048118 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:54.048093    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f911451a-2556-49ed-9778-7faa6549d341-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d797f569-bfs6w\" (UID: \"f911451a-2556-49ed-9778-7faa6549d341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d797f569-bfs6w"
Oct 30 19:29:54.049855 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:54.049835    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f911451a-2556-49ed-9778-7faa6549d341-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d797f569-bfs6w\" (UID: \"f911451a-2556-49ed-9778-7faa6549d341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d797f569-bfs6w"
Oct 30 19:29:54.055833 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:54.055812    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ktzv\" (UniqueName: \"kubernetes.io/projected/f911451a-2556-49ed-9778-7faa6549d341-kube-api-access-9ktzv\") pod \"maas-api-revoke-sub-d797f569-bfs6w\" (UID: \"f911451a-2556-49ed-9778-7faa6549d341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d797f569-bfs6w"
Oct 30 19:29:54.166410 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:54.166318    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d797f569-bfs6w"
Oct 30 19:29:54.469713 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:29:54.469675    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf911451a_2556_49ed_9778_7faa6549d341.slice/crio-f94212058099052b11aa4a51871faed7c9c40e0b5d30b022dd7ce8fde1662147 WatchSource:0}: Error finding container f94212058099052b11aa4a51871faed7c9c40e0b5d30b022dd7ce8fde1662147: Status 404 returned error can't find the container with id f94212058099052b11aa4a51871faed7c9c40e0b5d30b022dd7ce8fde1662147
Oct 30 19:29:54.471323 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:54.471294    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d797f569-bfs6w"]
Oct 30 19:29:54.539722 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:54.539691    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d797f569-bfs6w" event={"ID":"f911451a-2556-49ed-9778-7faa6549d341","Type":"ContainerStarted","Data":"f94212058099052b11aa4a51871faed7c9c40e0b5d30b022dd7ce8fde1662147"}
Oct 30 19:29:54.541375 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:54.541353    5043 generic.go:356] "Generic (PLEG): container finished" podID="52b13089-acb8-4eee-a734-04113162fca4" containerID="50ca142dc7057ad4e9c2e502e02d8f7ab60638ef8ae5fefb3f13c9073fb60278" exitCode=0
Oct 30 19:29:54.541493 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:54.541436    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a8a9a63-wh84h" event={"ID":"52b13089-acb8-4eee-a734-04113162fca4","Type":"ContainerDied","Data":"50ca142dc7057ad4e9c2e502e02d8f7ab60638ef8ae5fefb3f13c9073fb60278"}
Oct 30 19:29:54.542982 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:54.542960    5043 generic.go:356] "Generic (PLEG): container finished" podID="fb02228c-861e-41bb-b16e-31eb93072532" containerID="85ed113461217a29b2913b30b659dd369fe6d289b2b04b938d3215a61cde4466" exitCode=0
Oct 30 19:29:54.543061 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:54.543034    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8966f23c-llfvq" event={"ID":"fb02228c-861e-41bb-b16e-31eb93072532","Type":"ContainerDied","Data":"85ed113461217a29b2913b30b659dd369fe6d289b2b04b938d3215a61cde4466"}
Oct 30 19:29:54.980223 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:29:54.980188    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:29:55.554246 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:55.554160    5043 generic.go:356] "Generic (PLEG): container finished" podID="f911451a-2556-49ed-9778-7faa6549d341" containerID="73d3157577611f00bc912fdb50ebc5c22cf17d07e3f600e4c6983201fc5e9a08" exitCode=0
Oct 30 19:29:55.554684 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:55.554246    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d797f569-bfs6w" event={"ID":"f911451a-2556-49ed-9778-7faa6549d341","Type":"ContainerDied","Data":"73d3157577611f00bc912fdb50ebc5c22cf17d07e3f600e4c6983201fc5e9a08"}
Oct 30 19:29:55.795622 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:55.795588    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8966f23c-llfvq"
Oct 30 19:29:55.822358 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:55.822327    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a8a9a63-wh84h"
Oct 30 19:29:55.880533 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:55.880502    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb02228c-861e-41bb-b16e-31eb93072532-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb02228c-861e-41bb-b16e-31eb93072532-maas-api-xks-ca\") pod \"fb02228c-861e-41bb-b16e-31eb93072532\" (UID: \"fb02228c-861e-41bb-b16e-31eb93072532\") "
Oct 30 19:29:55.881255 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:55.881232    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52b13089-acb8-4eee-a734-04113162fca4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52b13089-acb8-4eee-a734-04113162fca4-maas-api-xks-ca\") pod \"52b13089-acb8-4eee-a734-04113162fca4\" (UID: \"52b13089-acb8-4eee-a734-04113162fca4\") "
Oct 30 19:29:55.881509 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:55.881490    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb02228c-861e-41bb-b16e-31eb93072532-kube-api-access-2kdgw\" (UniqueName: \"kubernetes.io/projected/fb02228c-861e-41bb-b16e-31eb93072532-kube-api-access-2kdgw\") pod \"fb02228c-861e-41bb-b16e-31eb93072532\" (UID: \"fb02228c-861e-41bb-b16e-31eb93072532\") "
Oct 30 19:29:55.881765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:55.881727    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52b13089-acb8-4eee-a734-04113162fca4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52b13089-acb8-4eee-a734-04113162fca4-maas-api-openshift-ca\") pod \"52b13089-acb8-4eee-a734-04113162fca4\" (UID: \"52b13089-acb8-4eee-a734-04113162fca4\") "
Oct 30 19:29:55.881941 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:55.881925    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb02228c-861e-41bb-b16e-31eb93072532-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb02228c-861e-41bb-b16e-31eb93072532-maas-api-openshift-ca\") pod \"fb02228c-861e-41bb-b16e-31eb93072532\" (UID: \"fb02228c-861e-41bb-b16e-31eb93072532\") "
Oct 30 19:29:55.882164 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:55.882147    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52b13089-acb8-4eee-a734-04113162fca4-kube-api-access-qkzt4\" (UniqueName: \"kubernetes.io/projected/52b13089-acb8-4eee-a734-04113162fca4-kube-api-access-qkzt4\") pod \"52b13089-acb8-4eee-a734-04113162fca4\" (UID: \"52b13089-acb8-4eee-a734-04113162fca4\") "
Oct 30 19:29:55.882512 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:55.882482    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb02228c-861e-41bb-b16e-31eb93072532-maas-api-openshift-ca" pod "fb02228c-861e-41bb-b16e-31eb93072532" (UID: "fb02228c-861e-41bb-b16e-31eb93072532"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:29:55.883002 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:55.882982    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb02228c-861e-41bb-b16e-31eb93072532-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:55.883351 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:55.883325    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52b13089-acb8-4eee-a734-04113162fca4-maas-api-openshift-ca" pod "52b13089-acb8-4eee-a734-04113162fca4" (UID: "52b13089-acb8-4eee-a734-04113162fca4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:29:55.886453 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:55.886424    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb02228c-861e-41bb-b16e-31eb93072532-kube-api-access-2kdgw" pod "fb02228c-861e-41bb-b16e-31eb93072532" (UID: "fb02228c-861e-41bb-b16e-31eb93072532"). InnerVolumeSpecName "kube-api-access-2kdgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:29:55.888780 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:55.888721    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb02228c-861e-41bb-b16e-31eb93072532-maas-api-xks-ca" pod "fb02228c-861e-41bb-b16e-31eb93072532" (UID: "fb02228c-861e-41bb-b16e-31eb93072532"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:29:55.889114 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:55.889081    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52b13089-acb8-4eee-a734-04113162fca4-maas-api-xks-ca" pod "52b13089-acb8-4eee-a734-04113162fca4" (UID: "52b13089-acb8-4eee-a734-04113162fca4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:29:55.890927 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:55.890895    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52b13089-acb8-4eee-a734-04113162fca4-kube-api-access-qkzt4" pod "52b13089-acb8-4eee-a734-04113162fca4" (UID: "52b13089-acb8-4eee-a734-04113162fca4"). InnerVolumeSpecName "kube-api-access-qkzt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:29:55.984180 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:55.984144    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52b13089-acb8-4eee-a734-04113162fca4-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:55.984180 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:55.984174    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kdgw\" (UniqueName: \"kubernetes.io/projected/fb02228c-861e-41bb-b16e-31eb93072532-kube-api-access-2kdgw\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:55.984180 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:55.984185    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52b13089-acb8-4eee-a734-04113162fca4-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:55.984462 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:55.984194    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkzt4\" (UniqueName: \"kubernetes.io/projected/52b13089-acb8-4eee-a734-04113162fca4-kube-api-access-qkzt4\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:55.984462 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:55.984203    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb02228c-861e-41bb-b16e-31eb93072532-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:56.568305 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:56.568270    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a8a9a63-wh84h"
Oct 30 19:29:56.568839 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:56.568411    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a8a9a63-wh84h" event={"ID":"52b13089-acb8-4eee-a734-04113162fca4","Type":"ContainerDied","Data":"43110050a10f0603e889ded23aaddc3dcbb0cf5152765e0c530759398328554b"}
Oct 30 19:29:56.568839 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:56.568448    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43110050a10f0603e889ded23aaddc3dcbb0cf5152765e0c530759398328554b"
Oct 30 19:29:56.570443 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:56.570420    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8966f23c-llfvq"
Oct 30 19:29:56.570569 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:56.570461    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8966f23c-llfvq" event={"ID":"fb02228c-861e-41bb-b16e-31eb93072532","Type":"ContainerDied","Data":"c226aec534e287807cacc71647828c92fd5f0a2ed2bb3788a77fefa686fce93e"}
Oct 30 19:29:56.570569 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:56.570511    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c226aec534e287807cacc71647828c92fd5f0a2ed2bb3788a77fefa686fce93e"
Oct 30 19:29:57.224101 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:57.214958    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d797f569-bfs6w"
Oct 30 19:29:57.408763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:57.407837    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f911451a-2556-49ed-9778-7faa6549d341-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f911451a-2556-49ed-9778-7faa6549d341-maas-api-xks-ca\") pod \"f911451a-2556-49ed-9778-7faa6549d341\" (UID: \"f911451a-2556-49ed-9778-7faa6549d341\") "
Oct 30 19:29:57.408763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:57.407984    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f911451a-2556-49ed-9778-7faa6549d341-kube-api-access-9ktzv\" (UniqueName: \"kubernetes.io/projected/f911451a-2556-49ed-9778-7faa6549d341-kube-api-access-9ktzv\") pod \"f911451a-2556-49ed-9778-7faa6549d341\" (UID: \"f911451a-2556-49ed-9778-7faa6549d341\") "
Oct 30 19:29:57.408763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:57.408254    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f911451a-2556-49ed-9778-7faa6549d341-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f911451a-2556-49ed-9778-7faa6549d341-maas-api-openshift-ca\") pod \"f911451a-2556-49ed-9778-7faa6549d341\" (UID: \"f911451a-2556-49ed-9778-7faa6549d341\") "
Oct 30 19:29:57.453711 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:57.453349    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f911451a-2556-49ed-9778-7faa6549d341-maas-api-openshift-ca" pod "f911451a-2556-49ed-9778-7faa6549d341" (UID: "f911451a-2556-49ed-9778-7faa6549d341"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:29:57.524767 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:57.524628    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f911451a-2556-49ed-9778-7faa6549d341-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:57.556216 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:57.556138    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f911451a-2556-49ed-9778-7faa6549d341-maas-api-xks-ca" pod "f911451a-2556-49ed-9778-7faa6549d341" (UID: "f911451a-2556-49ed-9778-7faa6549d341"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:29:57.677352 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:57.671526    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f911451a-2556-49ed-9778-7faa6549d341-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:57.715320 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:57.715279    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d797f569-bfs6w" event={"ID":"f911451a-2556-49ed-9778-7faa6549d341","Type":"ContainerDied","Data":"f94212058099052b11aa4a51871faed7c9c40e0b5d30b022dd7ce8fde1662147"}
Oct 30 19:29:57.715580 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:57.715561    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f94212058099052b11aa4a51871faed7c9c40e0b5d30b022dd7ce8fde1662147"
Oct 30 19:29:57.715774 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:57.715757    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d797f569-bfs6w"
Oct 30 19:29:57.755096 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:57.755057    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f911451a-2556-49ed-9778-7faa6549d341-kube-api-access-9ktzv" pod "f911451a-2556-49ed-9778-7faa6549d341" (UID: "f911451a-2556-49ed-9778-7faa6549d341"). InnerVolumeSpecName "kube-api-access-9ktzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:29:57.790760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:57.772649    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ktzv\" (UniqueName: \"kubernetes.io/projected/f911451a-2556-49ed-9778-7faa6549d341-kube-api-access-9ktzv\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:57.817328 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:57.817282    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:58.174874 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:58.174837    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:58.229362 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:58.229318    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:58.494677 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:58.494628    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded"
Oct 30 19:29:58.611807 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:58.601812    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:29:58.665761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:58.661876    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:58.678395 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:58.678356    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:58.834893 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:58.834793    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:59.013788 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:59.009632    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:29:59.253547 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:59.244270    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:29:59.253547 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:59.249846    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:59.253547 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:59.250019    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 30 19:29:59.282288 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:59.282240    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:29:59.292909 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:59.292861    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:59.296768 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:59.293574    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:29:59.486772 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:59.426342    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:59.486772 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:59.428046    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:29:59.509833 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:59.495861    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:59.509833 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:59.496276    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"
Oct 30 19:29:59.605763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:59.605220    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:59.605964 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:59.605944    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:29:59.748764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:59.687238    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:00.088897 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:59.774613    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:00.089492 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:59.852853    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:00.092821 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:59.900361    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:00.096759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:29:59.990376    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:30:00.100875 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:00.100849    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:30:00.115364 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:00.051427    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-6dg26" podUID="692c558e-cc23-4533-b6ad-0f982a82c187" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:00.140763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:00.053260    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:30:00.142102 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:00.142075    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:30:00.298771 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:00.245339    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:00.298771 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:00.274850    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:00.298771 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:00.285125    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"
Oct 30 19:30:00.320777 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:00.299292    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:00.320777 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:00.299679    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:30:00.320777 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:00.300021    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:30:00.441993 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:00.441891    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:00.511790 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:00.504074    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:00.820776 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:00.806505    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:30:00.981892 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:00.981851    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:30:01.109822 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:01.106844    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:01.109822 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:01.108254    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:30:01.199212 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:01.199169    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:30:01.403044 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:01.402943    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:01.455766 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:01.418541    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:30:01.471875 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:01.471825    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:01.500764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:01.475861    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:01.504910 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:01.504867    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:01.737825 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:01.712917    5043 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:30:01.737825 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:01.712996    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr" podUID="6251d2e1-9d28-4bb8-a667-f415d0f79df0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:01.808216 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:01.808159    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:02.134831 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:30:02.110036    5043 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.288s"
Oct 30 19:30:02.150926 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:02.150883    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-60577a-29889810-qhxp7"]
Oct 30 19:30:02.204782 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:02.204109    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:02.322796 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:02.299070    5043 cpu_manager.go:414] "containerMap: removing container" podUID="fb02228c-861e-41bb-b16e-31eb93072532" containerName="revoke-keys"
Oct 30 19:30:02.327142 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:02.327111    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb02228c-861e-41bb-b16e-31eb93072532" containerName="revoke-keys"
Oct 30 19:30:02.327438 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:02.327404    5043 cpu_manager.go:414] "containerMap: removing container" podUID="f911451a-2556-49ed-9778-7faa6549d341" containerName="revoke-keys"
Oct 30 19:30:02.330938 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:02.330904    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f911451a-2556-49ed-9778-7faa6549d341" containerName="revoke-keys"
Oct 30 19:30:02.351330 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:02.351302    5043 cpu_manager.go:414] "containerMap: removing container" podUID="52b13089-acb8-4eee-a734-04113162fca4" containerName="revoke-keys"
Oct 30 19:30:02.351556 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:02.351538    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52b13089-acb8-4eee-a734-04113162fca4" containerName="revoke-keys"
Oct 30 19:30:02.369762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:02.352166    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="52b13089-acb8-4eee-a734-04113162fca4" containerName="revoke-keys"
Oct 30 19:30:02.378891 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:02.378858    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb02228c-861e-41bb-b16e-31eb93072532" containerName="revoke-keys"
Oct 30 19:30:02.379152 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:02.379132    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="f911451a-2556-49ed-9778-7faa6549d341" containerName="revoke-keys"
Oct 30 19:30:02.614762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:02.593332    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889810-mlbkl"]
Oct 30 19:30:02.954543 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:02.954439    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:30:02.962092 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:02.962050    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:02.969774 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:02.966270    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:30:03.064643 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:02.956151    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:30:03.133837 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:02.970025    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:30:03.133837 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:02.668907    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:30:03.133837 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:02.956113    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:30:03.135150 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:02.661826    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-60577a-29889810-qhxp7"
Oct 30 19:30:03.365765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:03.249200    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:03.365765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:03.004859    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:03.365765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:03.168999    5043 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:30:03.365765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:03.249974    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9" podUID="224648b8-553f-44ad-9d9a-3784a7bea898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:03.365765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:03.215534    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:03.365765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:03.141293    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10f83fb1-b345-4e8d-bc3f-8e9ca4622d32-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-60577a-29889810-qhxp7\" (UID: \"10f83fb1-b345-4e8d-bc3f-8e9ca4622d32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-60577a-29889810-qhxp7"
Oct 30 19:30:03.365765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:03.250230    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fchp5\" (UniqueName: \"kubernetes.io/projected/10f83fb1-b345-4e8d-bc3f-8e9ca4622d32-kube-api-access-fchp5\") pod \"maas-api-key-cleanup-e2e-worker-w5-60577a-29889810-qhxp7\" (UID: \"10f83fb1-b345-4e8d-bc3f-8e9ca4622d32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-60577a-29889810-qhxp7"
Oct 30 19:30:03.365765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:03.316212    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:30:03.365765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:03.322923    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:30:03.494766 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:03.408098    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded"
Oct 30 19:30:03.582821 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:03.443280    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:30:03.582821 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:03.470190    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b623ca-29889810-mpsnp"]
Oct 30 19:30:03.600520 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:03.471586    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889810-mlbkl"
Oct 30 19:30:03.685994 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:03.685886    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:03.712337 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:03.712303    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10f83fb1-b345-4e8d-bc3f-8e9ca4622d32-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-60577a-29889810-qhxp7\" (UID: \"10f83fb1-b345-4e8d-bc3f-8e9ca4622d32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-60577a-29889810-qhxp7"
Oct 30 19:30:03.712768 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:03.712719    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fchp5\" (UniqueName: \"kubernetes.io/projected/10f83fb1-b345-4e8d-bc3f-8e9ca4622d32-kube-api-access-fchp5\") pod \"maas-api-key-cleanup-e2e-worker-w5-60577a-29889810-qhxp7\" (UID: \"10f83fb1-b345-4e8d-bc3f-8e9ca4622d32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-60577a-29889810-qhxp7"
Oct 30 19:30:03.713327 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:03.713295    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc4lj\" (UniqueName: \"kubernetes.io/projected/3b445eac-bb73-4f34-b7eb-8c7164fdfcf1-kube-api-access-fc4lj\") pod \"maas-api-key-cleanup-e2e-worker-w1-419d0a-29889810-mlbkl\" (UID: \"3b445eac-bb73-4f34-b7eb-8c7164fdfcf1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889810-mlbkl"
Oct 30 19:30:03.713694 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:03.713675    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b445eac-bb73-4f34-b7eb-8c7164fdfcf1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-419d0a-29889810-mlbkl\" (UID: \"3b445eac-bb73-4f34-b7eb-8c7164fdfcf1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889810-mlbkl"
Oct 30 19:30:03.730663 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:03.730616    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10f83fb1-b345-4e8d-bc3f-8e9ca4622d32-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-60577a-29889810-qhxp7\" (UID: \"10f83fb1-b345-4e8d-bc3f-8e9ca4622d32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-60577a-29889810-qhxp7"
Oct 30 19:30:03.898773 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:03.853602    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889810-kvb5s"]
Oct 30 19:30:03.925125 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:03.854573    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b623ca-29889810-mpsnp"
Oct 30 19:30:03.925811 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:03.878159    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fchp5\" (UniqueName: \"kubernetes.io/projected/10f83fb1-b345-4e8d-bc3f-8e9ca4622d32-kube-api-access-fchp5\") pod \"maas-api-key-cleanup-e2e-worker-w5-60577a-29889810-qhxp7\" (UID: \"10f83fb1-b345-4e8d-bc3f-8e9ca4622d32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-60577a-29889810-qhxp7"
Oct 30 19:30:03.925993 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:03.891665    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:03.972861 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:03.972822    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:03.973250 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:03.973221    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:30:04.050796 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:04.024543    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc4lj\" (UniqueName: \"kubernetes.io/projected/3b445eac-bb73-4f34-b7eb-8c7164fdfcf1-kube-api-access-fc4lj\") pod \"maas-api-key-cleanup-e2e-worker-w1-419d0a-29889810-mlbkl\" (UID: \"3b445eac-bb73-4f34-b7eb-8c7164fdfcf1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889810-mlbkl"
Oct 30 19:30:04.050796 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:04.024665    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b445eac-bb73-4f34-b7eb-8c7164fdfcf1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-419d0a-29889810-mlbkl\" (UID: \"3b445eac-bb73-4f34-b7eb-8c7164fdfcf1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889810-mlbkl"
Oct 30 19:30:04.050796 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:04.049983    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b445eac-bb73-4f34-b7eb-8c7164fdfcf1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-419d0a-29889810-mlbkl\" (UID: \"3b445eac-bb73-4f34-b7eb-8c7164fdfcf1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889810-mlbkl"
Oct 30 19:30:04.085784 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:04.074622    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-60577a-29889810-qhxp7"
Oct 30 19:30:04.131785 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:04.131400    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:30:04.216516 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:04.142222    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/036e429a-c9b2-4c90-8a6f-6af6ee075954-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b623ca-29889810-mpsnp\" (UID: \"036e429a-c9b2-4c90-8a6f-6af6ee075954\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b623ca-29889810-mpsnp"
Oct 30 19:30:04.279667 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:04.279496    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phvrh\" (UniqueName: \"kubernetes.io/projected/036e429a-c9b2-4c90-8a6f-6af6ee075954-kube-api-access-phvrh\") pod \"maas-api-key-cleanup-e2e-worker-w2-b623ca-29889810-mpsnp\" (UID: \"036e429a-c9b2-4c90-8a6f-6af6ee075954\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b623ca-29889810-mpsnp"
Oct 30 19:30:04.585463 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:04.143875    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:30:04.585463 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:04.143896    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:30:04.585463 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:04.143912    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:30:04.585463 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:04.143836    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:04.585463 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:04.305861    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:04.585463 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:04.306611    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:30:04.585463 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:04.333753    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc4lj\" (UniqueName: \"kubernetes.io/projected/3b445eac-bb73-4f34-b7eb-8c7164fdfcf1-kube-api-access-fc4lj\") pod \"maas-api-key-cleanup-e2e-worker-w1-419d0a-29889810-mlbkl\" (UID: \"3b445eac-bb73-4f34-b7eb-8c7164fdfcf1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889810-mlbkl"
Oct 30 19:30:04.585463 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:04.390997    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ae63bf-29889810-c65gt"]
Oct 30 19:30:04.760582 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:04.664194    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phvrh\" (UniqueName: \"kubernetes.io/projected/036e429a-c9b2-4c90-8a6f-6af6ee075954-kube-api-access-phvrh\") pod \"maas-api-key-cleanup-e2e-worker-w2-b623ca-29889810-mpsnp\" (UID: \"036e429a-c9b2-4c90-8a6f-6af6ee075954\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b623ca-29889810-mpsnp"
Oct 30 19:30:04.760582 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:04.664381    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/036e429a-c9b2-4c90-8a6f-6af6ee075954-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b623ca-29889810-mpsnp\" (UID: \"036e429a-c9b2-4c90-8a6f-6af6ee075954\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b623ca-29889810-mpsnp"
Oct 30 19:30:04.760582 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:04.664426    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c644e491-b875-42ea-b0ec-164b96a17176-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889810-kvb5s\" (UID: \"c644e491-b875-42ea-b0ec-164b96a17176\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889810-kvb5s"
Oct 30 19:30:04.760582 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:04.664513    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv9nk\" (UniqueName: \"kubernetes.io/projected/c644e491-b875-42ea-b0ec-164b96a17176-kube-api-access-jv9nk\") pod \"maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889810-kvb5s\" (UID: \"c644e491-b875-42ea-b0ec-164b96a17176\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889810-kvb5s"
Oct 30 19:30:04.760582 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:04.470353    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": context deadline exceeded"
Oct 30 19:30:04.760582 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:04.391098    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889810-kvb5s"
Oct 30 19:30:04.760582 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:04.690524    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/036e429a-c9b2-4c90-8a6f-6af6ee075954-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b623ca-29889810-mpsnp\" (UID: \"036e429a-c9b2-4c90-8a6f-6af6ee075954\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b623ca-29889810-mpsnp"
Oct 30 19:30:04.760582 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:04.541901    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:04.857806 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:04.804399    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889810-mlbkl"
Oct 30 19:30:04.930757 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:04.904656    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phvrh\" (UniqueName: \"kubernetes.io/projected/036e429a-c9b2-4c90-8a6f-6af6ee075954-kube-api-access-phvrh\") pod \"maas-api-key-cleanup-e2e-worker-w2-b623ca-29889810-mpsnp\" (UID: \"036e429a-c9b2-4c90-8a6f-6af6ee075954\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b623ca-29889810-mpsnp"
Oct 30 19:30:04.931569 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:04.920997    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889810-kvb5s"]
Oct 30 19:30:04.932060 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:04.932040    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b623ca-29889810-mpsnp"]
Oct 30 19:30:05.065600 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:04.996983    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:05.065600 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:04.997350    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 30 19:30:05.065600 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:04.921091    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ae63bf-29889810-c65gt"
Oct 30 19:30:05.185362 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:05.184169    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:05.252730 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:05.252691    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b623ca-29889810-mpsnp"
Oct 30 19:30:05.270934 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:05.270893    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05f835e2-2b71-40c2-ac2e-d8256c71f71c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ae63bf-29889810-c65gt\" (UID: \"05f835e2-2b71-40c2-ac2e-d8256c71f71c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ae63bf-29889810-c65gt"
Oct 30 19:30:05.271426 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:05.271399    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c644e491-b875-42ea-b0ec-164b96a17176-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889810-kvb5s\" (UID: \"c644e491-b875-42ea-b0ec-164b96a17176\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889810-kvb5s"
Oct 30 19:30:05.271629 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:05.271598    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv9nk\" (UniqueName: \"kubernetes.io/projected/c644e491-b875-42ea-b0ec-164b96a17176-kube-api-access-jv9nk\") pod \"maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889810-kvb5s\" (UID: \"c644e491-b875-42ea-b0ec-164b96a17176\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889810-kvb5s"
Oct 30 19:30:05.271830 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:05.271805    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzxrs\" (UniqueName: \"kubernetes.io/projected/05f835e2-2b71-40c2-ac2e-d8256c71f71c-kube-api-access-fzxrs\") pod \"maas-api-key-cleanup-e2e-worker-w0-ae63bf-29889810-c65gt\" (UID: \"05f835e2-2b71-40c2-ac2e-d8256c71f71c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ae63bf-29889810-c65gt"
Oct 30 19:30:05.277812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:05.276250    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c644e491-b875-42ea-b0ec-164b96a17176-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889810-kvb5s\" (UID: \"c644e491-b875-42ea-b0ec-164b96a17176\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889810-kvb5s"
Oct 30 19:30:05.505949 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:05.505919    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:30:05.506178 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:05.506166    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889810-mlbkl"]
Oct 30 19:30:05.506299 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:05.506288    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-60577a-29889810-qhxp7"]
Oct 30 19:30:05.506466 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:05.506452    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"
Oct 30 19:30:05.506603 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:05.506591    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:30:05.506722 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:05.506710    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ae63bf-29889810-c65gt"]
Oct 30 19:30:05.510048 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:05.509964    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:30:05.540702 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:05.539570    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzxrs\" (UniqueName: \"kubernetes.io/projected/05f835e2-2b71-40c2-ac2e-d8256c71f71c-kube-api-access-fzxrs\") pod \"maas-api-key-cleanup-e2e-worker-w0-ae63bf-29889810-c65gt\" (UID: \"05f835e2-2b71-40c2-ac2e-d8256c71f71c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ae63bf-29889810-c65gt"
Oct 30 19:30:05.540702 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:05.539688    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05f835e2-2b71-40c2-ac2e-d8256c71f71c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ae63bf-29889810-c65gt\" (UID: \"05f835e2-2b71-40c2-ac2e-d8256c71f71c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ae63bf-29889810-c65gt"
Oct 30 19:30:05.540946 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:05.540771    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05f835e2-2b71-40c2-ac2e-d8256c71f71c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ae63bf-29889810-c65gt\" (UID: \"05f835e2-2b71-40c2-ac2e-d8256c71f71c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ae63bf-29889810-c65gt"
Oct 30 19:30:05.543147 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:05.543115    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:05.549315 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:05.546806    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:30:05.563460 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:05.563005    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv9nk\" (UniqueName: \"kubernetes.io/projected/c644e491-b875-42ea-b0ec-164b96a17176-kube-api-access-jv9nk\") pod \"maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889810-kvb5s\" (UID: \"c644e491-b875-42ea-b0ec-164b96a17176\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889810-kvb5s"
Oct 30 19:30:05.587533 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:05.564405    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:05.587533 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:05.564845    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:05.587533 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:05.570591    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"
Oct 30 19:30:05.590534 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:05.590510    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:30:05.592637 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:05.592615    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889810-kvb5s"
Oct 30 19:30:05.601170 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:05.601142    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzxrs\" (UniqueName: \"kubernetes.io/projected/05f835e2-2b71-40c2-ac2e-d8256c71f71c-kube-api-access-fzxrs\") pod \"maas-api-key-cleanup-e2e-worker-w0-ae63bf-29889810-c65gt\" (UID: \"05f835e2-2b71-40c2-ac2e-d8256c71f71c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ae63bf-29889810-c65gt"
Oct 30 19:30:05.607170 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:05.607145    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:30:05.615809 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:05.615790    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:30:05.621541 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:05.621519    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:30:05.804158 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:05.804128    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d9567a8-229rl"]
Oct 30 19:30:05.811766 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:05.811727    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9567a8-229rl"
Oct 30 19:30:05.813025 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:05.812998    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d9567a8-229rl"]
Oct 30 19:30:05.821185 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:05.821115    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:30:05.847250 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:05.847219    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ae63bf-29889810-c65gt"
Oct 30 19:30:05.854349 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:05.854219    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f97nr\" (UniqueName: \"kubernetes.io/projected/d89aa71b-2ecb-475d-999f-7ed12d08098c-kube-api-access-f97nr\") pod \"maas-api-revoke-sub-3d9567a8-229rl\" (UID: \"d89aa71b-2ecb-475d-999f-7ed12d08098c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9567a8-229rl"
Oct 30 19:30:05.854349 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:05.854318    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d89aa71b-2ecb-475d-999f-7ed12d08098c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d9567a8-229rl\" (UID: \"d89aa71b-2ecb-475d-999f-7ed12d08098c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9567a8-229rl"
Oct 30 19:30:05.854809 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:05.854581    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d89aa71b-2ecb-475d-999f-7ed12d08098c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d9567a8-229rl\" (UID: \"d89aa71b-2ecb-475d-999f-7ed12d08098c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9567a8-229rl"
Oct 30 19:30:05.883686 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:30:05.883648    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f83fb1_b345_4e8d_bc3f_8e9ca4622d32.slice/crio-a31fbc43947c31081edcb12ee8af03825dfc914a13e857cb70bcefe1b3960d58 WatchSource:0}: Error finding container a31fbc43947c31081edcb12ee8af03825dfc914a13e857cb70bcefe1b3960d58: Status 404 returned error can't find the container with id a31fbc43947c31081edcb12ee8af03825dfc914a13e857cb70bcefe1b3960d58
Oct 30 19:30:05.935926 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:05.935320    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-60577a-29889810-qhxp7"]
Oct 30 19:30:05.956387 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:05.956282    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f97nr\" (UniqueName: \"kubernetes.io/projected/d89aa71b-2ecb-475d-999f-7ed12d08098c-kube-api-access-f97nr\") pod \"maas-api-revoke-sub-3d9567a8-229rl\" (UID: \"d89aa71b-2ecb-475d-999f-7ed12d08098c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9567a8-229rl"
Oct 30 19:30:05.957378 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:05.956617    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d89aa71b-2ecb-475d-999f-7ed12d08098c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d9567a8-229rl\" (UID: \"d89aa71b-2ecb-475d-999f-7ed12d08098c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9567a8-229rl"
Oct 30 19:30:05.957378 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:05.956842    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d89aa71b-2ecb-475d-999f-7ed12d08098c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d9567a8-229rl\" (UID: \"d89aa71b-2ecb-475d-999f-7ed12d08098c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9567a8-229rl"
Oct 30 19:30:05.960070 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:05.958527    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d89aa71b-2ecb-475d-999f-7ed12d08098c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d9567a8-229rl\" (UID: \"d89aa71b-2ecb-475d-999f-7ed12d08098c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9567a8-229rl"
Oct 30 19:30:05.960896 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:05.960411    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d89aa71b-2ecb-475d-999f-7ed12d08098c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d9567a8-229rl\" (UID: \"d89aa71b-2ecb-475d-999f-7ed12d08098c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9567a8-229rl"
Oct 30 19:30:06.000794 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:05.998936    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f97nr\" (UniqueName: \"kubernetes.io/projected/d89aa71b-2ecb-475d-999f-7ed12d08098c-kube-api-access-f97nr\") pod \"maas-api-revoke-sub-3d9567a8-229rl\" (UID: \"d89aa71b-2ecb-475d-999f-7ed12d08098c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9567a8-229rl"
Oct 30 19:30:06.141036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:06.141004    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9567a8-229rl"
Oct 30 19:30:06.146162 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:30:06.145607    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod036e429a_c9b2_4c90_8a6f_6af6ee075954.slice/crio-69cc69490768fa2885a1dbf23717aa97a9cb5459c9c59cbba9a917c41d7776f8 WatchSource:0}: Error finding container 69cc69490768fa2885a1dbf23717aa97a9cb5459c9c59cbba9a917c41d7776f8: Status 404 returned error can't find the container with id 69cc69490768fa2885a1dbf23717aa97a9cb5459c9c59cbba9a917c41d7776f8
Oct 30 19:30:06.151886 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:06.151858    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b623ca-29889810-mpsnp"]
Oct 30 19:30:06.163332 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:06.163299    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889810-mlbkl"]
Oct 30 19:30:06.177596 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:06.177568    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ae63bf-29889810-c65gt"]
Oct 30 19:30:06.186880 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:30:06.186710    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05f835e2_2b71_40c2_ac2e_d8256c71f71c.slice/crio-8e14bff91945b668539c7c13b66b5d52634ee3260b02902825f6bf578c59e2d0 WatchSource:0}: Error finding container 8e14bff91945b668539c7c13b66b5d52634ee3260b02902825f6bf578c59e2d0: Status 404 returned error can't find the container with id 8e14bff91945b668539c7c13b66b5d52634ee3260b02902825f6bf578c59e2d0
Oct 30 19:30:06.195188 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:06.195159    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889810-kvb5s"]
Oct 30 19:30:06.198610 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:30:06.198476    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc644e491_b875_42ea_b0ec_164b96a17176.slice/crio-1be56a0b8bf5200a0ea5b26759e950525266dd53a6cab5b41f583e910487ed6f WatchSource:0}: Error finding container 1be56a0b8bf5200a0ea5b26759e950525266dd53a6cab5b41f583e910487ed6f: Status 404 returned error can't find the container with id 1be56a0b8bf5200a0ea5b26759e950525266dd53a6cab5b41f583e910487ed6f
Oct 30 19:30:06.462455 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:30:06.462415    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd89aa71b_2ecb_475d_999f_7ed12d08098c.slice/crio-7e0fa70cf05768f7758e2cc502dfdb4664d7571ead79f202ae40fb49aae54299 WatchSource:0}: Error finding container 7e0fa70cf05768f7758e2cc502dfdb4664d7571ead79f202ae40fb49aae54299: Status 404 returned error can't find the container with id 7e0fa70cf05768f7758e2cc502dfdb4664d7571ead79f202ae40fb49aae54299
Oct 30 19:30:06.464091 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:06.464063    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d9567a8-229rl"]
Oct 30 19:30:06.619681 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:06.619643    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-60577a-29889810-qhxp7" event={"ID":"10f83fb1-b345-4e8d-bc3f-8e9ca4622d32","Type":"ContainerStarted","Data":"d823f3d41cb342ea4aa118ae1bd487798c0cd92e893cf46daf6323a99dc5512e"}
Oct 30 19:30:06.619876 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:06.619702    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-60577a-29889810-qhxp7" event={"ID":"10f83fb1-b345-4e8d-bc3f-8e9ca4622d32","Type":"ContainerStarted","Data":"a31fbc43947c31081edcb12ee8af03825dfc914a13e857cb70bcefe1b3960d58"}
Oct 30 19:30:06.621126 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:06.621102    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889810-kvb5s" event={"ID":"c644e491-b875-42ea-b0ec-164b96a17176","Type":"ContainerStarted","Data":"1be56a0b8bf5200a0ea5b26759e950525266dd53a6cab5b41f583e910487ed6f"}
Oct 30 19:30:06.622457 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:06.622439    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b623ca-29889810-mpsnp" event={"ID":"036e429a-c9b2-4c90-8a6f-6af6ee075954","Type":"ContainerStarted","Data":"69cc69490768fa2885a1dbf23717aa97a9cb5459c9c59cbba9a917c41d7776f8"}
Oct 30 19:30:06.624460 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:06.624432    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9567a8-229rl" event={"ID":"d89aa71b-2ecb-475d-999f-7ed12d08098c","Type":"ContainerStarted","Data":"7e0fa70cf05768f7758e2cc502dfdb4664d7571ead79f202ae40fb49aae54299"}
Oct 30 19:30:06.629369 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:06.629341    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ae63bf-29889810-c65gt" event={"ID":"05f835e2-2b71-40c2-ac2e-d8256c71f71c","Type":"ContainerStarted","Data":"8e14bff91945b668539c7c13b66b5d52634ee3260b02902825f6bf578c59e2d0"}
Oct 30 19:30:06.631054 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:06.631034    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889810-mlbkl" event={"ID":"3b445eac-bb73-4f34-b7eb-8c7164fdfcf1","Type":"ContainerStarted","Data":"c2c3fa9a86e7c28b5c1ad7b8f421891e9fc422103ce13ab5a48b5b6e38a5e43a"}
Oct 30 19:30:06.631141 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:06.631067    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889810-mlbkl" event={"ID":"3b445eac-bb73-4f34-b7eb-8c7164fdfcf1","Type":"ContainerStarted","Data":"2de3659da51932534764aa68ef8ef5faff2be4ccccf45de209a63ed2ef0e0b69"}
Oct 30 19:30:06.641645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:06.641560    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-60577a-29889810-qhxp7" podStartSLOduration=6.641544634 podStartE2EDuration="6.641544634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:06.634865325 +0000 UTC m=+826.145925302" watchObservedRunningTime="2026-10-30 19:30:06.641544634 +0000 UTC m=+826.152604606"
Oct 30 19:30:06.658126 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:06.658057    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889810-mlbkl" podStartSLOduration=6.658037512 podStartE2EDuration="6.658037512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:06.650584773 +0000 UTC m=+826.161644739" watchObservedRunningTime="2026-10-30 19:30:06.658037512 +0000 UTC m=+826.169097478"
Oct 30 19:30:07.641711 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:07.641620    5043 generic.go:356] "Generic (PLEG): container finished" podID="3b445eac-bb73-4f34-b7eb-8c7164fdfcf1" containerID="c2c3fa9a86e7c28b5c1ad7b8f421891e9fc422103ce13ab5a48b5b6e38a5e43a" exitCode=60
Oct 30 19:30:07.641711 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:07.641699    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889810-mlbkl" event={"ID":"3b445eac-bb73-4f34-b7eb-8c7164fdfcf1","Type":"ContainerDied","Data":"c2c3fa9a86e7c28b5c1ad7b8f421891e9fc422103ce13ab5a48b5b6e38a5e43a"}
Oct 30 19:30:07.642152 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:07.641917    5043 scope.go:122] "RemoveContainer" containerID="c2c3fa9a86e7c28b5c1ad7b8f421891e9fc422103ce13ab5a48b5b6e38a5e43a"
Oct 30 19:30:07.643501 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:07.643341    5043 generic.go:356] "Generic (PLEG): container finished" podID="10f83fb1-b345-4e8d-bc3f-8e9ca4622d32" containerID="d823f3d41cb342ea4aa118ae1bd487798c0cd92e893cf46daf6323a99dc5512e" exitCode=60
Oct 30 19:30:07.643501 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:07.643413    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-60577a-29889810-qhxp7" event={"ID":"10f83fb1-b345-4e8d-bc3f-8e9ca4622d32","Type":"ContainerDied","Data":"d823f3d41cb342ea4aa118ae1bd487798c0cd92e893cf46daf6323a99dc5512e"}
Oct 30 19:30:07.643668 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:07.643648    5043 scope.go:122] "RemoveContainer" containerID="d823f3d41cb342ea4aa118ae1bd487798c0cd92e893cf46daf6323a99dc5512e"
Oct 30 19:30:07.645461 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:07.645276    5043 generic.go:356] "Generic (PLEG): container finished" podID="c644e491-b875-42ea-b0ec-164b96a17176" containerID="2318b5419605a0626719ff7a30fe11d37839ee9500985f565d0780cb28035360" exitCode=60
Oct 30 19:30:07.645461 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:07.645382    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889810-kvb5s" event={"ID":"c644e491-b875-42ea-b0ec-164b96a17176","Type":"ContainerDied","Data":"2318b5419605a0626719ff7a30fe11d37839ee9500985f565d0780cb28035360"}
Oct 30 19:30:07.648315 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:07.646011    5043 scope.go:122] "RemoveContainer" containerID="2318b5419605a0626719ff7a30fe11d37839ee9500985f565d0780cb28035360"
Oct 30 19:30:07.650372 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:07.650347    5043 generic.go:356] "Generic (PLEG): container finished" podID="036e429a-c9b2-4c90-8a6f-6af6ee075954" containerID="ad727080eab7b7d47f35f2954743a036eb5b131743ae710161e0ac4fbf33416c" exitCode=60
Oct 30 19:30:07.650485 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:07.650444    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b623ca-29889810-mpsnp" event={"ID":"036e429a-c9b2-4c90-8a6f-6af6ee075954","Type":"ContainerDied","Data":"ad727080eab7b7d47f35f2954743a036eb5b131743ae710161e0ac4fbf33416c"}
Oct 30 19:30:07.650642 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:07.650622    5043 scope.go:122] "RemoveContainer" containerID="ad727080eab7b7d47f35f2954743a036eb5b131743ae710161e0ac4fbf33416c"
Oct 30 19:30:07.652779 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:07.652760    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9567a8-229rl" event={"ID":"d89aa71b-2ecb-475d-999f-7ed12d08098c","Type":"ContainerStarted","Data":"c0ad122ccd60265b8490ac4aef2015d653970d4ed114b26d3dd42fb77d0ab564"}
Oct 30 19:30:07.654652 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:07.654630    5043 generic.go:356] "Generic (PLEG): container finished" podID="05f835e2-2b71-40c2-ac2e-d8256c71f71c" containerID="7776edaf169b1a654244d81c485551d7a5a77c26db0197155881d988c6976b00" exitCode=60
Oct 30 19:30:07.654763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:07.654701    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ae63bf-29889810-c65gt" event={"ID":"05f835e2-2b71-40c2-ac2e-d8256c71f71c","Type":"ContainerDied","Data":"7776edaf169b1a654244d81c485551d7a5a77c26db0197155881d988c6976b00"}
Oct 30 19:30:07.654986 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:07.654972    5043 scope.go:122] "RemoveContainer" containerID="7776edaf169b1a654244d81c485551d7a5a77c26db0197155881d988c6976b00"
Oct 30 19:30:07.683314 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:07.683259    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9567a8-229rl" podStartSLOduration=4.683240008 podStartE2EDuration="4.683240008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:07.682635131 +0000 UTC m=+827.193695096" watchObservedRunningTime="2026-10-30 19:30:07.683240008 +0000 UTC m=+827.194299975"
Oct 30 19:30:08.025269 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:08.025236    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1383da43-dlbh5"]
Oct 30 19:30:08.031143 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:08.031116    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1383da43-dlbh5"
Oct 30 19:30:08.037591 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:08.037567    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1383da43-dlbh5"]
Oct 30 19:30:08.114581 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:08.114555    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e70ab44b-6364-4eca-8fb3-c0598ed37cf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1383da43-dlbh5\" (UID: \"e70ab44b-6364-4eca-8fb3-c0598ed37cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1383da43-dlbh5"
Oct 30 19:30:08.114722 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:08.114706    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e70ab44b-6364-4eca-8fb3-c0598ed37cf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1383da43-dlbh5\" (UID: \"e70ab44b-6364-4eca-8fb3-c0598ed37cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1383da43-dlbh5"
Oct 30 19:30:08.114814 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:08.114782    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6bx7\" (UniqueName: \"kubernetes.io/projected/e70ab44b-6364-4eca-8fb3-c0598ed37cf0-kube-api-access-z6bx7\") pod \"maas-api-revoke-sub-1383da43-dlbh5\" (UID: \"e70ab44b-6364-4eca-8fb3-c0598ed37cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1383da43-dlbh5"
Oct 30 19:30:08.217233 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:08.217201    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e70ab44b-6364-4eca-8fb3-c0598ed37cf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1383da43-dlbh5\" (UID: \"e70ab44b-6364-4eca-8fb3-c0598ed37cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1383da43-dlbh5"
Oct 30 19:30:08.217419 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:08.217320    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e70ab44b-6364-4eca-8fb3-c0598ed37cf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1383da43-dlbh5\" (UID: \"e70ab44b-6364-4eca-8fb3-c0598ed37cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1383da43-dlbh5"
Oct 30 19:30:08.217419 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:08.217350    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6bx7\" (UniqueName: \"kubernetes.io/projected/e70ab44b-6364-4eca-8fb3-c0598ed37cf0-kube-api-access-z6bx7\") pod \"maas-api-revoke-sub-1383da43-dlbh5\" (UID: \"e70ab44b-6364-4eca-8fb3-c0598ed37cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1383da43-dlbh5"
Oct 30 19:30:08.218025 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:08.217997    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e70ab44b-6364-4eca-8fb3-c0598ed37cf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1383da43-dlbh5\" (UID: \"e70ab44b-6364-4eca-8fb3-c0598ed37cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1383da43-dlbh5"
Oct 30 19:30:08.219791 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:08.219769    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e70ab44b-6364-4eca-8fb3-c0598ed37cf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1383da43-dlbh5\" (UID: \"e70ab44b-6364-4eca-8fb3-c0598ed37cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1383da43-dlbh5"
Oct 30 19:30:08.226135 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:08.226111    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6bx7\" (UniqueName: \"kubernetes.io/projected/e70ab44b-6364-4eca-8fb3-c0598ed37cf0-kube-api-access-z6bx7\") pod \"maas-api-revoke-sub-1383da43-dlbh5\" (UID: \"e70ab44b-6364-4eca-8fb3-c0598ed37cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1383da43-dlbh5"
Oct 30 19:30:08.351229 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:08.351195    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1383da43-dlbh5"
Oct 30 19:30:08.717138 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:08.717104    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889810-kvb5s" event={"ID":"c644e491-b875-42ea-b0ec-164b96a17176","Type":"ContainerStarted","Data":"f7959879e0331a73ea3f0853ff5cc4d2a49f8410a72952f42d3bb793026a58d6"}
Oct 30 19:30:08.720380 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:30:08.720352    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode70ab44b_6364_4eca_8fb3_c0598ed37cf0.slice/crio-3db16af025520a67b8209b9ad628824d3e480101f7d868c609670c78f0703e2c WatchSource:0}: Error finding container 3db16af025520a67b8209b9ad628824d3e480101f7d868c609670c78f0703e2c: Status 404 returned error can't find the container with id 3db16af025520a67b8209b9ad628824d3e480101f7d868c609670c78f0703e2c
Oct 30 19:30:08.721279 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:08.721124    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1383da43-dlbh5"]
Oct 30 19:30:08.723783 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:08.721845    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b623ca-29889810-mpsnp" event={"ID":"036e429a-c9b2-4c90-8a6f-6af6ee075954","Type":"ContainerStarted","Data":"6a7710d7a432726822218830f9118a9700be4814f6a680661c081bb978bbcf57"}
Oct 30 19:30:08.723783 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:08.722212    5043 scope.go:122] "RemoveContainer" containerID="6a7710d7a432726822218830f9118a9700be4814f6a680661c081bb978bbcf57"
Oct 30 19:30:08.723783 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:30:08.722541    5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b623ca-29889810-mpsnp_odh-ai-gateway-infra(036e429a-c9b2-4c90-8a6f-6af6ee075954)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b623ca-29889810-mpsnp" podUID="036e429a-c9b2-4c90-8a6f-6af6ee075954"
Oct 30 19:30:08.724196 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:08.724176    5043 generic.go:356] "Generic (PLEG): container finished" podID="d89aa71b-2ecb-475d-999f-7ed12d08098c" containerID="c0ad122ccd60265b8490ac4aef2015d653970d4ed114b26d3dd42fb77d0ab564" exitCode=0
Oct 30 19:30:08.724433 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:08.724413    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9567a8-229rl" event={"ID":"d89aa71b-2ecb-475d-999f-7ed12d08098c","Type":"ContainerDied","Data":"c0ad122ccd60265b8490ac4aef2015d653970d4ed114b26d3dd42fb77d0ab564"}
Oct 30 19:30:08.731538 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:08.731428    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ae63bf-29889810-c65gt" event={"ID":"05f835e2-2b71-40c2-ac2e-d8256c71f71c","Type":"ContainerStarted","Data":"5f2e39e010e80dab372d785cb65dc4dd3fb787a62f74ac8e34eb8f38edc63cc3"}
Oct 30 19:30:08.751239 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:08.751154    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889810-mlbkl" event={"ID":"3b445eac-bb73-4f34-b7eb-8c7164fdfcf1","Type":"ContainerStarted","Data":"16ac106d7818e40b234b328247356100d1812e6bb35254776c26ade0b2321b99"}
Oct 30 19:30:08.754045 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:08.754019    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-60577a-29889810-qhxp7" event={"ID":"10f83fb1-b345-4e8d-bc3f-8e9ca4622d32","Type":"ContainerStarted","Data":"c8d48427276af79a69014c019351daa9e71b8f70e7677607a0474aef39a455a8"}
Oct 30 19:30:08.754339 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:08.754288    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889810-kvb5s" podStartSLOduration=8.754269387 podStartE2EDuration="8.754269387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:08.748890238 +0000 UTC m=+828.259950216" watchObservedRunningTime="2026-10-30 19:30:08.754269387 +0000 UTC m=+828.265329353"
Oct 30 19:30:08.816254 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:08.816203    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ae63bf-29889810-c65gt" podStartSLOduration=8.816189128 podStartE2EDuration="8.816189128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:08.811161529 +0000 UTC m=+828.322221493" watchObservedRunningTime="2026-10-30 19:30:08.816189128 +0000 UTC m=+828.327249139"
Oct 30 19:30:09.770160 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:09.770121    5043 generic.go:356] "Generic (PLEG): container finished" podID="05f835e2-2b71-40c2-ac2e-d8256c71f71c" containerID="5f2e39e010e80dab372d785cb65dc4dd3fb787a62f74ac8e34eb8f38edc63cc3" exitCode=60
Oct 30 19:30:09.770594 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:09.770200    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ae63bf-29889810-c65gt" event={"ID":"05f835e2-2b71-40c2-ac2e-d8256c71f71c","Type":"ContainerDied","Data":"5f2e39e010e80dab372d785cb65dc4dd3fb787a62f74ac8e34eb8f38edc63cc3"}
Oct 30 19:30:09.770594 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:09.770260    5043 scope.go:122] "RemoveContainer" containerID="7776edaf169b1a654244d81c485551d7a5a77c26db0197155881d988c6976b00"
Oct 30 19:30:09.770691 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:09.770594    5043 scope.go:122] "RemoveContainer" containerID="5f2e39e010e80dab372d785cb65dc4dd3fb787a62f74ac8e34eb8f38edc63cc3"
Oct 30 19:30:09.771026 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:30:09.770998    5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ae63bf-29889810-c65gt_odh-ai-gateway-infra(05f835e2-2b71-40c2-ac2e-d8256c71f71c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ae63bf-29889810-c65gt" podUID="05f835e2-2b71-40c2-ac2e-d8256c71f71c"
Oct 30 19:30:09.773228 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:09.773207    5043 generic.go:356] "Generic (PLEG): container finished" podID="3b445eac-bb73-4f34-b7eb-8c7164fdfcf1" containerID="16ac106d7818e40b234b328247356100d1812e6bb35254776c26ade0b2321b99" exitCode=60
Oct 30 19:30:09.773461 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:09.773370    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889810-mlbkl" event={"ID":"3b445eac-bb73-4f34-b7eb-8c7164fdfcf1","Type":"ContainerDied","Data":"16ac106d7818e40b234b328247356100d1812e6bb35254776c26ade0b2321b99"}
Oct 30 19:30:09.774211 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:09.773803    5043 scope.go:122] "RemoveContainer" containerID="16ac106d7818e40b234b328247356100d1812e6bb35254776c26ade0b2321b99"
Oct 30 19:30:09.774211 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:30:09.774167    5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-419d0a-29889810-mlbkl_odh-ai-gateway-infra(3b445eac-bb73-4f34-b7eb-8c7164fdfcf1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889810-mlbkl" podUID="3b445eac-bb73-4f34-b7eb-8c7164fdfcf1"
Oct 30 19:30:09.776443 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:09.776398    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1383da43-dlbh5" event={"ID":"e70ab44b-6364-4eca-8fb3-c0598ed37cf0","Type":"ContainerStarted","Data":"f2ae8dfe297ea4a4abf4ad0228d55f2bdf54f44c5658e35f94afb802ab565450"}
Oct 30 19:30:09.776548 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:09.776453    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1383da43-dlbh5" event={"ID":"e70ab44b-6364-4eca-8fb3-c0598ed37cf0","Type":"ContainerStarted","Data":"3db16af025520a67b8209b9ad628824d3e480101f7d868c609670c78f0703e2c"}
Oct 30 19:30:09.778983 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:09.778963    5043 generic.go:356] "Generic (PLEG): container finished" podID="10f83fb1-b345-4e8d-bc3f-8e9ca4622d32" containerID="c8d48427276af79a69014c019351daa9e71b8f70e7677607a0474aef39a455a8" exitCode=60
Oct 30 19:30:09.779256 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:09.779226    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-60577a-29889810-qhxp7" event={"ID":"10f83fb1-b345-4e8d-bc3f-8e9ca4622d32","Type":"ContainerDied","Data":"c8d48427276af79a69014c019351daa9e71b8f70e7677607a0474aef39a455a8"}
Oct 30 19:30:09.779593 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:09.779576    5043 scope.go:122] "RemoveContainer" containerID="c8d48427276af79a69014c019351daa9e71b8f70e7677607a0474aef39a455a8"
Oct 30 19:30:09.779978 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:30:09.779956    5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-60577a-29889810-qhxp7_odh-ai-gateway-infra(10f83fb1-b345-4e8d-bc3f-8e9ca4622d32)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-60577a-29889810-qhxp7" podUID="10f83fb1-b345-4e8d-bc3f-8e9ca4622d32"
Oct 30 19:30:09.781675 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:09.781652    5043 generic.go:356] "Generic (PLEG): container finished" podID="c644e491-b875-42ea-b0ec-164b96a17176" containerID="f7959879e0331a73ea3f0853ff5cc4d2a49f8410a72952f42d3bb793026a58d6" exitCode=60
Oct 30 19:30:09.781798 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:09.781772    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889810-kvb5s" event={"ID":"c644e491-b875-42ea-b0ec-164b96a17176","Type":"ContainerDied","Data":"f7959879e0331a73ea3f0853ff5cc4d2a49f8410a72952f42d3bb793026a58d6"}
Oct 30 19:30:09.782022 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:09.782001    5043 scope.go:122] "RemoveContainer" containerID="f7959879e0331a73ea3f0853ff5cc4d2a49f8410a72952f42d3bb793026a58d6"
Oct 30 19:30:09.782298 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:30:09.782275    5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889810-kvb5s_odh-ai-gateway-infra(c644e491-b875-42ea-b0ec-164b96a17176)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889810-kvb5s" podUID="c644e491-b875-42ea-b0ec-164b96a17176"
Oct 30 19:30:09.784611 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:09.784448    5043 generic.go:356] "Generic (PLEG): container finished" podID="036e429a-c9b2-4c90-8a6f-6af6ee075954" containerID="6a7710d7a432726822218830f9118a9700be4814f6a680661c081bb978bbcf57" exitCode=60
Oct 30 19:30:09.784969 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:09.784947    5043 scope.go:122] "RemoveContainer" containerID="6a7710d7a432726822218830f9118a9700be4814f6a680661c081bb978bbcf57"
Oct 30 19:30:09.785272 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:30:09.785204    5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b623ca-29889810-mpsnp_odh-ai-gateway-infra(036e429a-c9b2-4c90-8a6f-6af6ee075954)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b623ca-29889810-mpsnp" podUID="036e429a-c9b2-4c90-8a6f-6af6ee075954"
Oct 30 19:30:09.785272 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:09.785253    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b623ca-29889810-mpsnp" event={"ID":"036e429a-c9b2-4c90-8a6f-6af6ee075954","Type":"ContainerDied","Data":"6a7710d7a432726822218830f9118a9700be4814f6a680661c081bb978bbcf57"}
Oct 30 19:30:09.789884 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:09.789866    5043 scope.go:122] "RemoveContainer" containerID="c2c3fa9a86e7c28b5c1ad7b8f421891e9fc422103ce13ab5a48b5b6e38a5e43a"
Oct 30 19:30:09.809725 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:09.809468    5043 scope.go:122] "RemoveContainer" containerID="d823f3d41cb342ea4aa118ae1bd487798c0cd92e893cf46daf6323a99dc5512e"
Oct 30 19:30:09.829701 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:09.829520    5043 scope.go:122] "RemoveContainer" containerID="2318b5419605a0626719ff7a30fe11d37839ee9500985f565d0780cb28035360"
Oct 30 19:30:09.846908 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:09.846884    5043 scope.go:122] "RemoveContainer" containerID="ad727080eab7b7d47f35f2954743a036eb5b131743ae710161e0ac4fbf33416c"
Oct 30 19:30:09.896749 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:09.882568    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1383da43-dlbh5" podStartSLOduration=1.882550648 podStartE2EDuration="1.882550648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:09.88071014 +0000 UTC m=+829.391770108" watchObservedRunningTime="2026-10-30 19:30:09.882550648 +0000 UTC m=+829.393610614"
Oct 30 19:30:10.015918 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:10.015891    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9567a8-229rl"
Oct 30 19:30:10.049924 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:10.049831    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d89aa71b-2ecb-475d-999f-7ed12d08098c-kube-api-access-f97nr\" (UniqueName: \"kubernetes.io/projected/d89aa71b-2ecb-475d-999f-7ed12d08098c-kube-api-access-f97nr\") pod \"d89aa71b-2ecb-475d-999f-7ed12d08098c\" (UID: \"d89aa71b-2ecb-475d-999f-7ed12d08098c\") "
Oct 30 19:30:10.050085 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:10.049927    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d89aa71b-2ecb-475d-999f-7ed12d08098c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d89aa71b-2ecb-475d-999f-7ed12d08098c-maas-api-openshift-ca\") pod \"d89aa71b-2ecb-475d-999f-7ed12d08098c\" (UID: \"d89aa71b-2ecb-475d-999f-7ed12d08098c\") "
Oct 30 19:30:10.050293 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:10.050272    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d89aa71b-2ecb-475d-999f-7ed12d08098c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d89aa71b-2ecb-475d-999f-7ed12d08098c-maas-api-xks-ca\") pod \"d89aa71b-2ecb-475d-999f-7ed12d08098c\" (UID: \"d89aa71b-2ecb-475d-999f-7ed12d08098c\") "
Oct 30 19:30:10.050374 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:10.050313    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d89aa71b-2ecb-475d-999f-7ed12d08098c-maas-api-openshift-ca" pod "d89aa71b-2ecb-475d-999f-7ed12d08098c" (UID: "d89aa71b-2ecb-475d-999f-7ed12d08098c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:10.051019 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:10.050993    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d89aa71b-2ecb-475d-999f-7ed12d08098c-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:10.052346 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:10.052304    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d89aa71b-2ecb-475d-999f-7ed12d08098c-kube-api-access-f97nr" pod "d89aa71b-2ecb-475d-999f-7ed12d08098c" (UID: "d89aa71b-2ecb-475d-999f-7ed12d08098c"). InnerVolumeSpecName "kube-api-access-f97nr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:10.053147 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:10.053121    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d89aa71b-2ecb-475d-999f-7ed12d08098c-maas-api-xks-ca" pod "d89aa71b-2ecb-475d-999f-7ed12d08098c" (UID: "d89aa71b-2ecb-475d-999f-7ed12d08098c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:30:10.152633 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:10.152595    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d89aa71b-2ecb-475d-999f-7ed12d08098c-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:10.152633 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:10.152633    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f97nr\" (UniqueName: \"kubernetes.io/projected/d89aa71b-2ecb-475d-999f-7ed12d08098c-kube-api-access-f97nr\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:10.797388 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:10.797352    5043 scope.go:122] "RemoveContainer" containerID="c8d48427276af79a69014c019351daa9e71b8f70e7677607a0474aef39a455a8"
Oct 30 19:30:10.797844 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:30:10.797723    5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-60577a-29889810-qhxp7_odh-ai-gateway-infra(10f83fb1-b345-4e8d-bc3f-8e9ca4622d32)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-60577a-29889810-qhxp7" podUID="10f83fb1-b345-4e8d-bc3f-8e9ca4622d32"
Oct 30 19:30:10.799253 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:10.799233    5043 scope.go:122] "RemoveContainer" containerID="f7959879e0331a73ea3f0853ff5cc4d2a49f8410a72952f42d3bb793026a58d6"
Oct 30 19:30:10.799563 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:30:10.799536    5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889810-kvb5s_odh-ai-gateway-infra(c644e491-b875-42ea-b0ec-164b96a17176)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889810-kvb5s" podUID="c644e491-b875-42ea-b0ec-164b96a17176"
Oct 30 19:30:10.801193 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:10.801175    5043 scope.go:122] "RemoveContainer" containerID="6a7710d7a432726822218830f9118a9700be4814f6a680661c081bb978bbcf57"
Oct 30 19:30:10.801501 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:30:10.801481    5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b623ca-29889810-mpsnp_odh-ai-gateway-infra(036e429a-c9b2-4c90-8a6f-6af6ee075954)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b623ca-29889810-mpsnp" podUID="036e429a-c9b2-4c90-8a6f-6af6ee075954"
Oct 30 19:30:10.802329 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:10.802311    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9567a8-229rl"
Oct 30 19:30:10.802418 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:10.802346    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9567a8-229rl" event={"ID":"d89aa71b-2ecb-475d-999f-7ed12d08098c","Type":"ContainerDied","Data":"7e0fa70cf05768f7758e2cc502dfdb4664d7571ead79f202ae40fb49aae54299"}
Oct 30 19:30:10.802418 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:10.802391    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e0fa70cf05768f7758e2cc502dfdb4664d7571ead79f202ae40fb49aae54299"
Oct 30 19:30:10.804321 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:10.804301    5043 scope.go:122] "RemoveContainer" containerID="5f2e39e010e80dab372d785cb65dc4dd3fb787a62f74ac8e34eb8f38edc63cc3"
Oct 30 19:30:10.804575 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:30:10.804553    5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ae63bf-29889810-c65gt_odh-ai-gateway-infra(05f835e2-2b71-40c2-ac2e-d8256c71f71c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ae63bf-29889810-c65gt" podUID="05f835e2-2b71-40c2-ac2e-d8256c71f71c"
Oct 30 19:30:10.806560 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:10.806543    5043 scope.go:122] "RemoveContainer" containerID="16ac106d7818e40b234b328247356100d1812e6bb35254776c26ade0b2321b99"
Oct 30 19:30:10.806829 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:30:10.806799    5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-419d0a-29889810-mlbkl_odh-ai-gateway-infra(3b445eac-bb73-4f34-b7eb-8c7164fdfcf1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889810-mlbkl" podUID="3b445eac-bb73-4f34-b7eb-8c7164fdfcf1"
Oct 30 19:30:10.807959 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:10.807942    5043 generic.go:356] "Generic (PLEG): container finished" podID="e70ab44b-6364-4eca-8fb3-c0598ed37cf0" containerID="f2ae8dfe297ea4a4abf4ad0228d55f2bdf54f44c5658e35f94afb802ab565450" exitCode=0
Oct 30 19:30:10.808042 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:10.808025    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1383da43-dlbh5" event={"ID":"e70ab44b-6364-4eca-8fb3-c0598ed37cf0","Type":"ContainerDied","Data":"f2ae8dfe297ea4a4abf4ad0228d55f2bdf54f44c5658e35f94afb802ab565450"}
Oct 30 19:30:12.037389 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:12.037359    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1383da43-dlbh5"
Oct 30 19:30:12.090115 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:12.090023    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e70ab44b-6364-4eca-8fb3-c0598ed37cf0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e70ab44b-6364-4eca-8fb3-c0598ed37cf0-maas-api-xks-ca\") pod \"e70ab44b-6364-4eca-8fb3-c0598ed37cf0\" (UID: \"e70ab44b-6364-4eca-8fb3-c0598ed37cf0\") "
Oct 30 19:30:12.090260 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:12.090149    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e70ab44b-6364-4eca-8fb3-c0598ed37cf0-kube-api-access-z6bx7\" (UniqueName: \"kubernetes.io/projected/e70ab44b-6364-4eca-8fb3-c0598ed37cf0-kube-api-access-z6bx7\") pod \"e70ab44b-6364-4eca-8fb3-c0598ed37cf0\" (UID: \"e70ab44b-6364-4eca-8fb3-c0598ed37cf0\") "
Oct 30 19:30:12.090467 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:12.090442    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e70ab44b-6364-4eca-8fb3-c0598ed37cf0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e70ab44b-6364-4eca-8fb3-c0598ed37cf0-maas-api-openshift-ca\") pod \"e70ab44b-6364-4eca-8fb3-c0598ed37cf0\" (UID: \"e70ab44b-6364-4eca-8fb3-c0598ed37cf0\") "
Oct 30 19:30:12.091401 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:12.091370    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e70ab44b-6364-4eca-8fb3-c0598ed37cf0-maas-api-openshift-ca" pod "e70ab44b-6364-4eca-8fb3-c0598ed37cf0" (UID: "e70ab44b-6364-4eca-8fb3-c0598ed37cf0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:12.093270 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:12.093240    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e70ab44b-6364-4eca-8fb3-c0598ed37cf0-maas-api-xks-ca" pod "e70ab44b-6364-4eca-8fb3-c0598ed37cf0" (UID: "e70ab44b-6364-4eca-8fb3-c0598ed37cf0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:30:12.093392 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:12.093273    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e70ab44b-6364-4eca-8fb3-c0598ed37cf0-kube-api-access-z6bx7" pod "e70ab44b-6364-4eca-8fb3-c0598ed37cf0" (UID: "e70ab44b-6364-4eca-8fb3-c0598ed37cf0"). InnerVolumeSpecName "kube-api-access-z6bx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:12.192578 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:12.192535    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e70ab44b-6364-4eca-8fb3-c0598ed37cf0-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:12.192578 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:12.192564    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6bx7\" (UniqueName: \"kubernetes.io/projected/e70ab44b-6364-4eca-8fb3-c0598ed37cf0-kube-api-access-z6bx7\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:12.192578 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:12.192574    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e70ab44b-6364-4eca-8fb3-c0598ed37cf0-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:12.846190 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:12.846159    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1383da43-dlbh5"
Oct 30 19:30:12.846379 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:12.846191    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1383da43-dlbh5" event={"ID":"e70ab44b-6364-4eca-8fb3-c0598ed37cf0","Type":"ContainerDied","Data":"3db16af025520a67b8209b9ad628824d3e480101f7d868c609670c78f0703e2c"}
Oct 30 19:30:12.846379 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:12.846241    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3db16af025520a67b8209b9ad628824d3e480101f7d868c609670c78f0703e2c"
Oct 30 19:30:14.715156 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:14.715122    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec322b7e-4rvf8"]
Oct 30 19:30:14.715788 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:14.715770    5043 cpu_manager.go:414] "containerMap: removing container" podUID="d89aa71b-2ecb-475d-999f-7ed12d08098c" containerName="revoke-keys"
Oct 30 19:30:14.715788 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:14.715789    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d89aa71b-2ecb-475d-999f-7ed12d08098c" containerName="revoke-keys"
Oct 30 19:30:14.715881 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:14.715800    5043 cpu_manager.go:414] "containerMap: removing container" podUID="e70ab44b-6364-4eca-8fb3-c0598ed37cf0" containerName="revoke-keys"
Oct 30 19:30:14.715881 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:14.715805    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e70ab44b-6364-4eca-8fb3-c0598ed37cf0" containerName="revoke-keys"
Oct 30 19:30:14.715986 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:14.715947    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="d89aa71b-2ecb-475d-999f-7ed12d08098c" containerName="revoke-keys"
Oct 30 19:30:14.715986 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:14.715956    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="e70ab44b-6364-4eca-8fb3-c0598ed37cf0" containerName="revoke-keys"
Oct 30 19:30:14.720500 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:14.720478    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec322b7e-4rvf8"
Oct 30 19:30:14.727002 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:14.726978    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:30:14.735454 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:14.735427    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec322b7e-4rvf8"]
Oct 30 19:30:14.835890 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:14.835860    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e5a119b-b436-41dd-9d61-70095e33bb6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec322b7e-4rvf8\" (UID: \"7e5a119b-b436-41dd-9d61-70095e33bb6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec322b7e-4rvf8"
Oct 30 19:30:14.836069 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:14.835982    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e5a119b-b436-41dd-9d61-70095e33bb6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec322b7e-4rvf8\" (UID: \"7e5a119b-b436-41dd-9d61-70095e33bb6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec322b7e-4rvf8"
Oct 30 19:30:14.836069 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:14.836065    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq2md\" (UniqueName: \"kubernetes.io/projected/7e5a119b-b436-41dd-9d61-70095e33bb6f-kube-api-access-qq2md\") pod \"maas-api-revoke-sub-ec322b7e-4rvf8\" (UID: \"7e5a119b-b436-41dd-9d61-70095e33bb6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec322b7e-4rvf8"
Oct 30 19:30:14.937458 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:14.937424    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e5a119b-b436-41dd-9d61-70095e33bb6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec322b7e-4rvf8\" (UID: \"7e5a119b-b436-41dd-9d61-70095e33bb6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec322b7e-4rvf8"
Oct 30 19:30:14.937659 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:14.937506    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq2md\" (UniqueName: \"kubernetes.io/projected/7e5a119b-b436-41dd-9d61-70095e33bb6f-kube-api-access-qq2md\") pod \"maas-api-revoke-sub-ec322b7e-4rvf8\" (UID: \"7e5a119b-b436-41dd-9d61-70095e33bb6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec322b7e-4rvf8"
Oct 30 19:30:14.937780 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:14.937685    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e5a119b-b436-41dd-9d61-70095e33bb6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec322b7e-4rvf8\" (UID: \"7e5a119b-b436-41dd-9d61-70095e33bb6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec322b7e-4rvf8"
Oct 30 19:30:14.938226 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:14.938198    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e5a119b-b436-41dd-9d61-70095e33bb6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec322b7e-4rvf8\" (UID: \"7e5a119b-b436-41dd-9d61-70095e33bb6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec322b7e-4rvf8"
Oct 30 19:30:14.940368 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:14.940344    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e5a119b-b436-41dd-9d61-70095e33bb6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec322b7e-4rvf8\" (UID: \"7e5a119b-b436-41dd-9d61-70095e33bb6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec322b7e-4rvf8"
Oct 30 19:30:14.947395 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:14.947368    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq2md\" (UniqueName: \"kubernetes.io/projected/7e5a119b-b436-41dd-9d61-70095e33bb6f-kube-api-access-qq2md\") pod \"maas-api-revoke-sub-ec322b7e-4rvf8\" (UID: \"7e5a119b-b436-41dd-9d61-70095e33bb6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec322b7e-4rvf8"
Oct 30 19:30:15.038610 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:15.038522    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec322b7e-4rvf8"
Oct 30 19:30:15.336616 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:30:15.336583    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e5a119b_b436_41dd_9d61_70095e33bb6f.slice/crio-58e11bfaaee8281e020df0fb3ce204568aa9b4a87c5083e461784ab52cd4bcb1 WatchSource:0}: Error finding container 58e11bfaaee8281e020df0fb3ce204568aa9b4a87c5083e461784ab52cd4bcb1: Status 404 returned error can't find the container with id 58e11bfaaee8281e020df0fb3ce204568aa9b4a87c5083e461784ab52cd4bcb1
Oct 30 19:30:15.338879 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:15.338855    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec322b7e-4rvf8"]
Oct 30 19:30:15.873291 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:15.873257    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec322b7e-4rvf8" event={"ID":"7e5a119b-b436-41dd-9d61-70095e33bb6f","Type":"ContainerStarted","Data":"a09a74f4888b31d7f16381070f111e54a8a57610bcb588bb91c78bdca27dd91a"}
Oct 30 19:30:15.873709 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:15.873315    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec322b7e-4rvf8" event={"ID":"7e5a119b-b436-41dd-9d61-70095e33bb6f","Type":"ContainerStarted","Data":"58e11bfaaee8281e020df0fb3ce204568aa9b4a87c5083e461784ab52cd4bcb1"}
Oct 30 19:30:15.891096 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:15.891044    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec322b7e-4rvf8" podStartSLOduration=1.89103171 podStartE2EDuration="1.89103171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:15.889409624 +0000 UTC m=+835.400469591" watchObservedRunningTime="2026-10-30 19:30:15.89103171 +0000 UTC m=+835.402091675"
Oct 30 19:30:16.885818 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:16.885784    5043 generic.go:356] "Generic (PLEG): container finished" podID="7e5a119b-b436-41dd-9d61-70095e33bb6f" containerID="a09a74f4888b31d7f16381070f111e54a8a57610bcb588bb91c78bdca27dd91a" exitCode=0
Oct 30 19:30:16.886240 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:16.885874    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec322b7e-4rvf8" event={"ID":"7e5a119b-b436-41dd-9d61-70095e33bb6f","Type":"ContainerDied","Data":"a09a74f4888b31d7f16381070f111e54a8a57610bcb588bb91c78bdca27dd91a"}
Oct 30 19:30:18.095746 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:18.095699    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec322b7e-4rvf8"
Oct 30 19:30:18.192350 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:18.192263    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e5a119b-b436-41dd-9d61-70095e33bb6f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e5a119b-b436-41dd-9d61-70095e33bb6f-maas-api-openshift-ca\") pod \"7e5a119b-b436-41dd-9d61-70095e33bb6f\" (UID: \"7e5a119b-b436-41dd-9d61-70095e33bb6f\") "
Oct 30 19:30:18.192523 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:18.192376    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e5a119b-b436-41dd-9d61-70095e33bb6f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e5a119b-b436-41dd-9d61-70095e33bb6f-maas-api-xks-ca\") pod \"7e5a119b-b436-41dd-9d61-70095e33bb6f\" (UID: \"7e5a119b-b436-41dd-9d61-70095e33bb6f\") "
Oct 30 19:30:18.192615 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:18.192592    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e5a119b-b436-41dd-9d61-70095e33bb6f-kube-api-access-qq2md\" (UniqueName: \"kubernetes.io/projected/7e5a119b-b436-41dd-9d61-70095e33bb6f-kube-api-access-qq2md\") pod \"7e5a119b-b436-41dd-9d61-70095e33bb6f\" (UID: \"7e5a119b-b436-41dd-9d61-70095e33bb6f\") "
Oct 30 19:30:18.192672 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:18.192619    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e5a119b-b436-41dd-9d61-70095e33bb6f-maas-api-openshift-ca" pod "7e5a119b-b436-41dd-9d61-70095e33bb6f" (UID: "7e5a119b-b436-41dd-9d61-70095e33bb6f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:18.193413 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:18.193388    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e5a119b-b436-41dd-9d61-70095e33bb6f-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:18.195393 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:18.195359    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e5a119b-b436-41dd-9d61-70095e33bb6f-maas-api-xks-ca" pod "7e5a119b-b436-41dd-9d61-70095e33bb6f" (UID: "7e5a119b-b436-41dd-9d61-70095e33bb6f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:30:18.195507 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:18.195480    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e5a119b-b436-41dd-9d61-70095e33bb6f-kube-api-access-qq2md" pod "7e5a119b-b436-41dd-9d61-70095e33bb6f" (UID: "7e5a119b-b436-41dd-9d61-70095e33bb6f"). InnerVolumeSpecName "kube-api-access-qq2md". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:18.294836 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:18.294805    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e5a119b-b436-41dd-9d61-70095e33bb6f-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:18.294836 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:18.294834    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq2md\" (UniqueName: \"kubernetes.io/projected/7e5a119b-b436-41dd-9d61-70095e33bb6f-kube-api-access-qq2md\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:18.920036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:18.919938    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec322b7e-4rvf8" event={"ID":"7e5a119b-b436-41dd-9d61-70095e33bb6f","Type":"ContainerDied","Data":"58e11bfaaee8281e020df0fb3ce204568aa9b4a87c5083e461784ab52cd4bcb1"}
Oct 30 19:30:18.920217 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:18.920044    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58e11bfaaee8281e020df0fb3ce204568aa9b4a87c5083e461784ab52cd4bcb1"
Oct 30 19:30:18.920217 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:18.920151    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec322b7e-4rvf8"
Oct 30 19:30:19.638209 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:19.638183    5043 scope.go:122] "RemoveContainer" containerID="6a7710d7a432726822218830f9118a9700be4814f6a680661c081bb978bbcf57"
Oct 30 19:30:20.943457 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:20.943373    5043 generic.go:356] "Generic (PLEG): container finished" podID="036e429a-c9b2-4c90-8a6f-6af6ee075954" containerID="34a167d27696a5558cf1186abae3c701df4d2e5a5385f35a617244cac2817926" exitCode=60
Oct 30 19:30:20.943880 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:20.943446    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b623ca-29889810-mpsnp" event={"ID":"036e429a-c9b2-4c90-8a6f-6af6ee075954","Type":"ContainerDied","Data":"34a167d27696a5558cf1186abae3c701df4d2e5a5385f35a617244cac2817926"}
Oct 30 19:30:20.943880 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:20.943508    5043 scope.go:122] "RemoveContainer" containerID="6a7710d7a432726822218830f9118a9700be4814f6a680661c081bb978bbcf57"
Oct 30 19:30:20.943991 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:20.943959    5043 scope.go:122] "RemoveContainer" containerID="34a167d27696a5558cf1186abae3c701df4d2e5a5385f35a617244cac2817926"
Oct 30 19:30:20.944361 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:30:20.944329    5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b623ca-29889810-mpsnp_odh-ai-gateway-infra(036e429a-c9b2-4c90-8a6f-6af6ee075954)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b623ca-29889810-mpsnp" podUID="036e429a-c9b2-4c90-8a6f-6af6ee075954"
Oct 30 19:30:21.637537 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:21.637505    5043 scope.go:122] "RemoveContainer" containerID="5f2e39e010e80dab372d785cb65dc4dd3fb787a62f74ac8e34eb8f38edc63cc3"
Oct 30 19:30:21.977919 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:21.977880    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b623ca-29889810-mpsnp"]
Oct 30 19:30:22.164513 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:22.163217    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b623ca-29889810-mpsnp"
Oct 30 19:30:22.266086 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:22.265991    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/036e429a-c9b2-4c90-8a6f-6af6ee075954-kube-api-access-phvrh\" (UniqueName: \"kubernetes.io/projected/036e429a-c9b2-4c90-8a6f-6af6ee075954-kube-api-access-phvrh\") pod \"036e429a-c9b2-4c90-8a6f-6af6ee075954\" (UID: \"036e429a-c9b2-4c90-8a6f-6af6ee075954\") "
Oct 30 19:30:22.266778 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:22.266166    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/036e429a-c9b2-4c90-8a6f-6af6ee075954-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/036e429a-c9b2-4c90-8a6f-6af6ee075954-maas-api-service-ca\") pod \"036e429a-c9b2-4c90-8a6f-6af6ee075954\" (UID: \"036e429a-c9b2-4c90-8a6f-6af6ee075954\") "
Oct 30 19:30:22.267287 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:22.267252    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/036e429a-c9b2-4c90-8a6f-6af6ee075954-maas-api-service-ca" pod "036e429a-c9b2-4c90-8a6f-6af6ee075954" (UID: "036e429a-c9b2-4c90-8a6f-6af6ee075954"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:22.313954 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:22.313910    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/036e429a-c9b2-4c90-8a6f-6af6ee075954-kube-api-access-phvrh" pod "036e429a-c9b2-4c90-8a6f-6af6ee075954" (UID: "036e429a-c9b2-4c90-8a6f-6af6ee075954"). InnerVolumeSpecName "kube-api-access-phvrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:22.368774 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:22.368724    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phvrh\" (UniqueName: \"kubernetes.io/projected/036e429a-c9b2-4c90-8a6f-6af6ee075954-kube-api-access-phvrh\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:22.369016 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:22.368991    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/036e429a-c9b2-4c90-8a6f-6af6ee075954-maas-api-service-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:22.637911 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:22.637879    5043 scope.go:122] "RemoveContainer" containerID="c8d48427276af79a69014c019351daa9e71b8f70e7677607a0474aef39a455a8"
Oct 30 19:30:22.638214 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:22.638178    5043 scope.go:122] "RemoveContainer" containerID="f7959879e0331a73ea3f0853ff5cc4d2a49f8410a72952f42d3bb793026a58d6"
Oct 30 19:30:22.638310 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:22.638291    5043 scope.go:122] "RemoveContainer" containerID="16ac106d7818e40b234b328247356100d1812e6bb35254776c26ade0b2321b99"
Oct 30 19:30:22.968416 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:22.968384    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b623ca-29889810-mpsnp"
Oct 30 19:30:22.968416 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:22.968405    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b623ca-29889810-mpsnp" event={"ID":"036e429a-c9b2-4c90-8a6f-6af6ee075954","Type":"ContainerDied","Data":"69cc69490768fa2885a1dbf23717aa97a9cb5459c9c59cbba9a917c41d7776f8"}
Oct 30 19:30:22.968666 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:22.968470    5043 scope.go:122] "RemoveContainer" containerID="34a167d27696a5558cf1186abae3c701df4d2e5a5385f35a617244cac2817926"
Oct 30 19:30:22.973093 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:22.973060    5043 generic.go:356] "Generic (PLEG): container finished" podID="05f835e2-2b71-40c2-ac2e-d8256c71f71c" containerID="c0de641599997eb9683c54ac0191297709d200afa5bd5ab0f81fe772c960c8b2" exitCode=60
Oct 30 19:30:22.973229 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:22.973205    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ae63bf-29889810-c65gt" event={"ID":"05f835e2-2b71-40c2-ac2e-d8256c71f71c","Type":"ContainerDied","Data":"c0de641599997eb9683c54ac0191297709d200afa5bd5ab0f81fe772c960c8b2"}
Oct 30 19:30:22.973707 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:22.973689    5043 scope.go:122] "RemoveContainer" containerID="c0de641599997eb9683c54ac0191297709d200afa5bd5ab0f81fe772c960c8b2"
Oct 30 19:30:22.974172 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:30:22.974147    5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ae63bf-29889810-c65gt_odh-ai-gateway-infra(05f835e2-2b71-40c2-ac2e-d8256c71f71c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ae63bf-29889810-c65gt" podUID="05f835e2-2b71-40c2-ac2e-d8256c71f71c"
Oct 30 19:30:22.985583 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:22.985564    5043 scope.go:122] "RemoveContainer" containerID="5f2e39e010e80dab372d785cb65dc4dd3fb787a62f74ac8e34eb8f38edc63cc3"
Oct 30 19:30:23.007356 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:23.007331    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b623ca-29889810-mpsnp"]
Oct 30 19:30:23.015555 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:23.015530    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b623ca-29889810-mpsnp"]
Oct 30 19:30:23.987549 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:23.987512    5043 generic.go:356] "Generic (PLEG): container finished" podID="10f83fb1-b345-4e8d-bc3f-8e9ca4622d32" containerID="554918796b6180fc087019ab8ba83e7ddeccb53252e440bb2b479fe37872866c" exitCode=60
Oct 30 19:30:23.988065 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:23.987580    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-60577a-29889810-qhxp7" event={"ID":"10f83fb1-b345-4e8d-bc3f-8e9ca4622d32","Type":"ContainerDied","Data":"554918796b6180fc087019ab8ba83e7ddeccb53252e440bb2b479fe37872866c"}
Oct 30 19:30:23.988065 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:23.987642    5043 scope.go:122] "RemoveContainer" containerID="c8d48427276af79a69014c019351daa9e71b8f70e7677607a0474aef39a455a8"
Oct 30 19:30:23.988175 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:23.988102    5043 scope.go:122] "RemoveContainer" containerID="554918796b6180fc087019ab8ba83e7ddeccb53252e440bb2b479fe37872866c"
Oct 30 19:30:23.988507 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:30:23.988484    5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-60577a-29889810-qhxp7_odh-ai-gateway-infra(10f83fb1-b345-4e8d-bc3f-8e9ca4622d32)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-60577a-29889810-qhxp7" podUID="10f83fb1-b345-4e8d-bc3f-8e9ca4622d32"
Oct 30 19:30:23.990373 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:23.990351    5043 generic.go:356] "Generic (PLEG): container finished" podID="c644e491-b875-42ea-b0ec-164b96a17176" containerID="388020e4980cdb1772e9a3925b1f6033860e6cc5c09e3a1e9c9ea428872e7e73" exitCode=60
Oct 30 19:30:23.990500 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:23.990425    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889810-kvb5s" event={"ID":"c644e491-b875-42ea-b0ec-164b96a17176","Type":"ContainerDied","Data":"388020e4980cdb1772e9a3925b1f6033860e6cc5c09e3a1e9c9ea428872e7e73"}
Oct 30 19:30:23.990859 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:23.990839    5043 scope.go:122] "RemoveContainer" containerID="388020e4980cdb1772e9a3925b1f6033860e6cc5c09e3a1e9c9ea428872e7e73"
Oct 30 19:30:23.991200 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:30:23.991181    5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889810-kvb5s_odh-ai-gateway-infra(c644e491-b875-42ea-b0ec-164b96a17176)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889810-kvb5s" podUID="c644e491-b875-42ea-b0ec-164b96a17176"
Oct 30 19:30:23.996868 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:23.996847    5043 generic.go:356] "Generic (PLEG): container finished" podID="3b445eac-bb73-4f34-b7eb-8c7164fdfcf1" containerID="9991356b08299b8ed6e956a3e2d55cac3f1a3fce9b04d12479034d31dfc46af1" exitCode=60
Oct 30 19:30:23.996958 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:23.996920    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889810-mlbkl" event={"ID":"3b445eac-bb73-4f34-b7eb-8c7164fdfcf1","Type":"ContainerDied","Data":"9991356b08299b8ed6e956a3e2d55cac3f1a3fce9b04d12479034d31dfc46af1"}
Oct 30 19:30:23.997312 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:23.997296    5043 scope.go:122] "RemoveContainer" containerID="9991356b08299b8ed6e956a3e2d55cac3f1a3fce9b04d12479034d31dfc46af1"
Oct 30 19:30:23.997604 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:30:23.997583    5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-419d0a-29889810-mlbkl_odh-ai-gateway-infra(3b445eac-bb73-4f34-b7eb-8c7164fdfcf1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889810-mlbkl" podUID="3b445eac-bb73-4f34-b7eb-8c7164fdfcf1"
Oct 30 19:30:24.004535 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:24.004514    5043 scope.go:122] "RemoveContainer" containerID="f7959879e0331a73ea3f0853ff5cc4d2a49f8410a72952f42d3bb793026a58d6"
Oct 30 19:30:24.063058 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:24.063014    5043 scope.go:122] "RemoveContainer" containerID="16ac106d7818e40b234b328247356100d1812e6bb35254776c26ade0b2321b99"
Oct 30 19:30:24.076112 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:24.075180    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ae63bf-29889810-c65gt"]
Oct 30 19:30:24.299403 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:24.299378    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ae63bf-29889810-c65gt"
Oct 30 19:30:24.406675 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:24.406641    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05f835e2-2b71-40c2-ac2e-d8256c71f71c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05f835e2-2b71-40c2-ac2e-d8256c71f71c-maas-api-service-ca\") pod \"05f835e2-2b71-40c2-ac2e-d8256c71f71c\" (UID: \"05f835e2-2b71-40c2-ac2e-d8256c71f71c\") "
Oct 30 19:30:24.406880 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:24.406854    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05f835e2-2b71-40c2-ac2e-d8256c71f71c-kube-api-access-fzxrs\" (UniqueName: \"kubernetes.io/projected/05f835e2-2b71-40c2-ac2e-d8256c71f71c-kube-api-access-fzxrs\") pod \"05f835e2-2b71-40c2-ac2e-d8256c71f71c\" (UID: \"05f835e2-2b71-40c2-ac2e-d8256c71f71c\") "
Oct 30 19:30:24.407067 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:24.407042    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05f835e2-2b71-40c2-ac2e-d8256c71f71c-maas-api-service-ca" pod "05f835e2-2b71-40c2-ac2e-d8256c71f71c" (UID: "05f835e2-2b71-40c2-ac2e-d8256c71f71c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:24.407371 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:24.407355    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05f835e2-2b71-40c2-ac2e-d8256c71f71c-maas-api-service-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:24.409213 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:24.409190    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05f835e2-2b71-40c2-ac2e-d8256c71f71c-kube-api-access-fzxrs" pod "05f835e2-2b71-40c2-ac2e-d8256c71f71c" (UID: "05f835e2-2b71-40c2-ac2e-d8256c71f71c"). InnerVolumeSpecName "kube-api-access-fzxrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:24.501588 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:24.501555    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cc1938a-bblth"]
Oct 30 19:30:24.502302 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:24.502287    5043 cpu_manager.go:414] "containerMap: removing container" podUID="05f835e2-2b71-40c2-ac2e-d8256c71f71c" containerName="cleanup"
Oct 30 19:30:24.502371 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:24.502303    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05f835e2-2b71-40c2-ac2e-d8256c71f71c" containerName="cleanup"
Oct 30 19:30:24.502371 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:24.502319    5043 cpu_manager.go:414] "containerMap: removing container" podUID="05f835e2-2b71-40c2-ac2e-d8256c71f71c" containerName="cleanup"
Oct 30 19:30:24.502371 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:24.502325    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05f835e2-2b71-40c2-ac2e-d8256c71f71c" containerName="cleanup"
Oct 30 19:30:24.502371 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:24.502343    5043 cpu_manager.go:414] "containerMap: removing container" podUID="05f835e2-2b71-40c2-ac2e-d8256c71f71c" containerName="cleanup"
Oct 30 19:30:24.502371 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:24.502348    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05f835e2-2b71-40c2-ac2e-d8256c71f71c" containerName="cleanup"
Oct 30 19:30:24.502371 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:24.502360    5043 cpu_manager.go:414] "containerMap: removing container" podUID="036e429a-c9b2-4c90-8a6f-6af6ee075954" containerName="cleanup"
Oct 30 19:30:24.502371 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:24.502365    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="036e429a-c9b2-4c90-8a6f-6af6ee075954" containerName="cleanup"
Oct 30 19:30:24.502580 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:24.502376    5043 cpu_manager.go:414] "containerMap: removing container" podUID="036e429a-c9b2-4c90-8a6f-6af6ee075954" containerName="cleanup"
Oct 30 19:30:24.502580 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:24.502382    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="036e429a-c9b2-4c90-8a6f-6af6ee075954" containerName="cleanup"
Oct 30 19:30:24.502580 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:24.502397    5043 cpu_manager.go:414] "containerMap: removing container" podUID="036e429a-c9b2-4c90-8a6f-6af6ee075954" containerName="cleanup"
Oct 30 19:30:24.502580 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:24.502401    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="036e429a-c9b2-4c90-8a6f-6af6ee075954" containerName="cleanup"
Oct 30 19:30:24.502580 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:24.502419    5043 cpu_manager.go:414] "containerMap: removing container" podUID="7e5a119b-b436-41dd-9d61-70095e33bb6f" containerName="revoke-keys"
Oct 30 19:30:24.502580 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:24.502423    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e5a119b-b436-41dd-9d61-70095e33bb6f" containerName="revoke-keys"
Oct 30 19:30:24.502580 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:24.502525    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="036e429a-c9b2-4c90-8a6f-6af6ee075954" containerName="cleanup"
Oct 30 19:30:24.502580 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:24.502537    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e5a119b-b436-41dd-9d61-70095e33bb6f" containerName="revoke-keys"
Oct 30 19:30:24.502580 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:24.502546    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="05f835e2-2b71-40c2-ac2e-d8256c71f71c" containerName="cleanup"
Oct 30 19:30:24.502580 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:24.502552    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="05f835e2-2b71-40c2-ac2e-d8256c71f71c" containerName="cleanup"
Oct 30 19:30:24.502580 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:24.502560    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="05f835e2-2b71-40c2-ac2e-d8256c71f71c" containerName="cleanup"
Oct 30 19:30:24.502580 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:24.502566    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="036e429a-c9b2-4c90-8a6f-6af6ee075954" containerName="cleanup"
Oct 30 19:30:24.506338 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:24.506314    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc1938a-bblth"
Oct 30 19:30:24.509498 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:24.509466    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzxrs\" (UniqueName: \"kubernetes.io/projected/05f835e2-2b71-40c2-ac2e-d8256c71f71c-kube-api-access-fzxrs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:24.509984 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:24.509951    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:30:24.516509 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:24.516441    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cc1938a-bblth"]
Oct 30 19:30:24.611112 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:24.611068    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8451c454-50d0-45ef-b0c8-7b81f855e6db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cc1938a-bblth\" (UID: \"8451c454-50d0-45ef-b0c8-7b81f855e6db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc1938a-bblth"
Oct 30 19:30:24.611417 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:24.611392    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrs5n\" (UniqueName: \"kubernetes.io/projected/8451c454-50d0-45ef-b0c8-7b81f855e6db-kube-api-access-rrs5n\") pod \"maas-api-revoke-sub-8cc1938a-bblth\" (UID: \"8451c454-50d0-45ef-b0c8-7b81f855e6db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc1938a-bblth"
Oct 30 19:30:24.611529 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:24.611443    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8451c454-50d0-45ef-b0c8-7b81f855e6db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cc1938a-bblth\" (UID: \"8451c454-50d0-45ef-b0c8-7b81f855e6db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc1938a-bblth"
Oct 30 19:30:24.646135 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:24.646098    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="036e429a-c9b2-4c90-8a6f-6af6ee075954" path="/var/lib/kubelet/pods/036e429a-c9b2-4c90-8a6f-6af6ee075954/volumes"
Oct 30 19:30:24.712686 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:24.712653    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8451c454-50d0-45ef-b0c8-7b81f855e6db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cc1938a-bblth\" (UID: \"8451c454-50d0-45ef-b0c8-7b81f855e6db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc1938a-bblth"
Oct 30 19:30:24.712952 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:24.712931    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrs5n\" (UniqueName: \"kubernetes.io/projected/8451c454-50d0-45ef-b0c8-7b81f855e6db-kube-api-access-rrs5n\") pod \"maas-api-revoke-sub-8cc1938a-bblth\" (UID: \"8451c454-50d0-45ef-b0c8-7b81f855e6db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc1938a-bblth"
Oct 30 19:30:24.713016 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:24.712987    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8451c454-50d0-45ef-b0c8-7b81f855e6db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cc1938a-bblth\" (UID: \"8451c454-50d0-45ef-b0c8-7b81f855e6db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc1938a-bblth"
Oct 30 19:30:24.713379 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:24.713358    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8451c454-50d0-45ef-b0c8-7b81f855e6db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cc1938a-bblth\" (UID: \"8451c454-50d0-45ef-b0c8-7b81f855e6db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc1938a-bblth"
Oct 30 19:30:24.715485 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:24.715463    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8451c454-50d0-45ef-b0c8-7b81f855e6db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cc1938a-bblth\" (UID: \"8451c454-50d0-45ef-b0c8-7b81f855e6db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc1938a-bblth"
Oct 30 19:30:24.726670 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:24.726648    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrs5n\" (UniqueName: \"kubernetes.io/projected/8451c454-50d0-45ef-b0c8-7b81f855e6db-kube-api-access-rrs5n\") pod \"maas-api-revoke-sub-8cc1938a-bblth\" (UID: \"8451c454-50d0-45ef-b0c8-7b81f855e6db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc1938a-bblth"
Oct 30 19:30:24.824098 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:24.824070    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc1938a-bblth"
Oct 30 19:30:25.006840 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:30:24.998643    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:30:25.025409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:25.025373    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-60577a-29889810-qhxp7"]
Oct 30 19:30:25.033060 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:25.033038    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ae63bf-29889810-c65gt"
Oct 30 19:30:25.033177 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:25.033044    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ae63bf-29889810-c65gt" event={"ID":"05f835e2-2b71-40c2-ac2e-d8256c71f71c","Type":"ContainerDied","Data":"8e14bff91945b668539c7c13b66b5d52634ee3260b02902825f6bf578c59e2d0"}
Oct 30 19:30:25.033177 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:25.033173    5043 scope.go:122] "RemoveContainer" containerID="c0de641599997eb9683c54ac0191297709d200afa5bd5ab0f81fe772c960c8b2"
Oct 30 19:30:25.041958 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:25.041793    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889810-kvb5s"]
Oct 30 19:30:25.083785 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:25.081606    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ae63bf-29889810-c65gt"]
Oct 30 19:30:25.090472 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:25.090288    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ae63bf-29889810-c65gt"]
Oct 30 19:30:25.096086 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:25.096057    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889810-mlbkl"]
Oct 30 19:30:25.181723 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:25.164623    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cc1938a-bblth"]
Oct 30 19:30:25.244409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:25.243403    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-60577a-29889810-qhxp7"
Oct 30 19:30:25.332862 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:25.332826    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10f83fb1-b345-4e8d-bc3f-8e9ca4622d32-kube-api-access-fchp5\" (UniqueName: \"kubernetes.io/projected/10f83fb1-b345-4e8d-bc3f-8e9ca4622d32-kube-api-access-fchp5\") pod \"10f83fb1-b345-4e8d-bc3f-8e9ca4622d32\" (UID: \"10f83fb1-b345-4e8d-bc3f-8e9ca4622d32\") "
Oct 30 19:30:25.333093 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:25.333063    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10f83fb1-b345-4e8d-bc3f-8e9ca4622d32-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10f83fb1-b345-4e8d-bc3f-8e9ca4622d32-maas-api-service-ca\") pod \"10f83fb1-b345-4e8d-bc3f-8e9ca4622d32\" (UID: \"10f83fb1-b345-4e8d-bc3f-8e9ca4622d32\") "
Oct 30 19:30:25.334045 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:25.334012    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10f83fb1-b345-4e8d-bc3f-8e9ca4622d32-maas-api-service-ca" pod "10f83fb1-b345-4e8d-bc3f-8e9ca4622d32" (UID: "10f83fb1-b345-4e8d-bc3f-8e9ca4622d32"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:25.357683 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:25.357648    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10f83fb1-b345-4e8d-bc3f-8e9ca4622d32-kube-api-access-fchp5" pod "10f83fb1-b345-4e8d-bc3f-8e9ca4622d32" (UID: "10f83fb1-b345-4e8d-bc3f-8e9ca4622d32"). InnerVolumeSpecName "kube-api-access-fchp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:25.370592 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:25.370567    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889810-kvb5s"
Oct 30 19:30:25.378000 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:25.377979    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889810-mlbkl"
Oct 30 19:30:25.435108 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:25.435072    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b445eac-bb73-4f34-b7eb-8c7164fdfcf1-kube-api-access-fc4lj\" (UniqueName: \"kubernetes.io/projected/3b445eac-bb73-4f34-b7eb-8c7164fdfcf1-kube-api-access-fc4lj\") pod \"3b445eac-bb73-4f34-b7eb-8c7164fdfcf1\" (UID: \"3b445eac-bb73-4f34-b7eb-8c7164fdfcf1\") "
Oct 30 19:30:25.435287 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:25.435127    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c644e491-b875-42ea-b0ec-164b96a17176-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c644e491-b875-42ea-b0ec-164b96a17176-maas-api-service-ca\") pod \"c644e491-b875-42ea-b0ec-164b96a17176\" (UID: \"c644e491-b875-42ea-b0ec-164b96a17176\") "
Oct 30 19:30:25.435287 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:25.435259    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c644e491-b875-42ea-b0ec-164b96a17176-kube-api-access-jv9nk\" (UniqueName: \"kubernetes.io/projected/c644e491-b875-42ea-b0ec-164b96a17176-kube-api-access-jv9nk\") pod \"c644e491-b875-42ea-b0ec-164b96a17176\" (UID: \"c644e491-b875-42ea-b0ec-164b96a17176\") "
Oct 30 19:30:25.435503 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:25.435477    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b445eac-bb73-4f34-b7eb-8c7164fdfcf1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b445eac-bb73-4f34-b7eb-8c7164fdfcf1-maas-api-service-ca\") pod \"3b445eac-bb73-4f34-b7eb-8c7164fdfcf1\" (UID: \"3b445eac-bb73-4f34-b7eb-8c7164fdfcf1\") "
Oct 30 19:30:25.435581 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:25.435560    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c644e491-b875-42ea-b0ec-164b96a17176-maas-api-service-ca" pod "c644e491-b875-42ea-b0ec-164b96a17176" (UID: "c644e491-b875-42ea-b0ec-164b96a17176"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:25.436184 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:25.436060    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b445eac-bb73-4f34-b7eb-8c7164fdfcf1-maas-api-service-ca" pod "3b445eac-bb73-4f34-b7eb-8c7164fdfcf1" (UID: "3b445eac-bb73-4f34-b7eb-8c7164fdfcf1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:25.436300 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:25.436241    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10f83fb1-b345-4e8d-bc3f-8e9ca4622d32-maas-api-service-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:25.436300 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:25.436263    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fchp5\" (UniqueName: \"kubernetes.io/projected/10f83fb1-b345-4e8d-bc3f-8e9ca4622d32-kube-api-access-fchp5\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:25.436300 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:25.436278    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c644e491-b875-42ea-b0ec-164b96a17176-maas-api-service-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:25.437877 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:25.437852    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b445eac-bb73-4f34-b7eb-8c7164fdfcf1-kube-api-access-fc4lj" pod "3b445eac-bb73-4f34-b7eb-8c7164fdfcf1" (UID: "3b445eac-bb73-4f34-b7eb-8c7164fdfcf1"). InnerVolumeSpecName "kube-api-access-fc4lj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:25.438808 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:25.438781    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c644e491-b875-42ea-b0ec-164b96a17176-kube-api-access-jv9nk" pod "c644e491-b875-42ea-b0ec-164b96a17176" (UID: "c644e491-b875-42ea-b0ec-164b96a17176"). InnerVolumeSpecName "kube-api-access-jv9nk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:25.538168 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:25.538066    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b445eac-bb73-4f34-b7eb-8c7164fdfcf1-maas-api-service-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:25.538168 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:25.538090    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fc4lj\" (UniqueName: \"kubernetes.io/projected/3b445eac-bb73-4f34-b7eb-8c7164fdfcf1-kube-api-access-fc4lj\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:25.538168 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:25.538100    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv9nk\" (UniqueName: \"kubernetes.io/projected/c644e491-b875-42ea-b0ec-164b96a17176-kube-api-access-jv9nk\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:26.055848 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.055812    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889810-kvb5s"
Oct 30 19:30:26.056343 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.055893    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889810-kvb5s" event={"ID":"c644e491-b875-42ea-b0ec-164b96a17176","Type":"ContainerDied","Data":"1be56a0b8bf5200a0ea5b26759e950525266dd53a6cab5b41f583e910487ed6f"}
Oct 30 19:30:26.056343 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.055955    5043 scope.go:122] "RemoveContainer" containerID="388020e4980cdb1772e9a3925b1f6033860e6cc5c09e3a1e9c9ea428872e7e73"
Oct 30 19:30:26.060003 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.059981    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889810-mlbkl"
Oct 30 19:30:26.061376 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.061347    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889810-mlbkl" event={"ID":"3b445eac-bb73-4f34-b7eb-8c7164fdfcf1","Type":"ContainerDied","Data":"2de3659da51932534764aa68ef8ef5faff2be4ccccf45de209a63ed2ef0e0b69"}
Oct 30 19:30:26.065109 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.063986    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc1938a-bblth" event={"ID":"8451c454-50d0-45ef-b0c8-7b81f855e6db","Type":"ContainerStarted","Data":"573b34198874a4f72afaa00c9477052d3ac1c32dd1522437c3dba33b5b09a608"}
Oct 30 19:30:26.065109 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.064032    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc1938a-bblth" event={"ID":"8451c454-50d0-45ef-b0c8-7b81f855e6db","Type":"ContainerStarted","Data":"e90521784b19651c2e49be7a3d564144ecd2c4fc87cfbd60cf6489c84dc7590c"}
Oct 30 19:30:26.065992 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.065974    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-60577a-29889810-qhxp7"
Oct 30 19:30:26.066135 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.066115    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-60577a-29889810-qhxp7" event={"ID":"10f83fb1-b345-4e8d-bc3f-8e9ca4622d32","Type":"ContainerDied","Data":"a31fbc43947c31081edcb12ee8af03825dfc914a13e857cb70bcefe1b3960d58"}
Oct 30 19:30:26.074841 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.073269    5043 scope.go:122] "RemoveContainer" containerID="9991356b08299b8ed6e956a3e2d55cac3f1a3fce9b04d12479034d31dfc46af1"
Oct 30 19:30:26.089317 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.089297    5043 scope.go:122] "RemoveContainer" containerID="554918796b6180fc087019ab8ba83e7ddeccb53252e440bb2b479fe37872866c"
Oct 30 19:30:26.096815 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.096775    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc1938a-bblth" podStartSLOduration=2.09675951 podStartE2EDuration="2.09675951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:26.089119009 +0000 UTC m=+845.600178973" watchObservedRunningTime="2026-10-30 19:30:26.09675951 +0000 UTC m=+845.607819475"
Oct 30 19:30:26.132352 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.132323    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889810-mlbkl"]
Oct 30 19:30:26.138368 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.138346    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889810-mlbkl"]
Oct 30 19:30:26.148853 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.148832    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-60577a-29889810-qhxp7"]
Oct 30 19:30:26.154130 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.154107    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-60577a-29889810-qhxp7"]
Oct 30 19:30:26.165273 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.165248    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889810-kvb5s"]
Oct 30 19:30:26.173463 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.173435    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889810-kvb5s"]
Oct 30 19:30:26.542760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.534398    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-235d91c2-bnzkl"]
Oct 30 19:30:26.542760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.535433    5043 cpu_manager.go:414] "containerMap: removing container" podUID="3b445eac-bb73-4f34-b7eb-8c7164fdfcf1" containerName="cleanup"
Oct 30 19:30:26.542760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.535452    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b445eac-bb73-4f34-b7eb-8c7164fdfcf1" containerName="cleanup"
Oct 30 19:30:26.542760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.535467    5043 cpu_manager.go:414] "containerMap: removing container" podUID="c644e491-b875-42ea-b0ec-164b96a17176" containerName="cleanup"
Oct 30 19:30:26.542760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.535474    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c644e491-b875-42ea-b0ec-164b96a17176" containerName="cleanup"
Oct 30 19:30:26.542760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.535502    5043 cpu_manager.go:414] "containerMap: removing container" podUID="10f83fb1-b345-4e8d-bc3f-8e9ca4622d32" containerName="cleanup"
Oct 30 19:30:26.542760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.535511    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10f83fb1-b345-4e8d-bc3f-8e9ca4622d32" containerName="cleanup"
Oct 30 19:30:26.542760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.535522    5043 cpu_manager.go:414] "containerMap: removing container" podUID="c644e491-b875-42ea-b0ec-164b96a17176" containerName="cleanup"
Oct 30 19:30:26.542760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.535529    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c644e491-b875-42ea-b0ec-164b96a17176" containerName="cleanup"
Oct 30 19:30:26.542760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.535540    5043 cpu_manager.go:414] "containerMap: removing container" podUID="10f83fb1-b345-4e8d-bc3f-8e9ca4622d32" containerName="cleanup"
Oct 30 19:30:26.542760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.535547    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10f83fb1-b345-4e8d-bc3f-8e9ca4622d32" containerName="cleanup"
Oct 30 19:30:26.542760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.535571    5043 cpu_manager.go:414] "containerMap: removing container" podUID="3b445eac-bb73-4f34-b7eb-8c7164fdfcf1" containerName="cleanup"
Oct 30 19:30:26.542760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.535577    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b445eac-bb73-4f34-b7eb-8c7164fdfcf1" containerName="cleanup"
Oct 30 19:30:26.542760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.535598    5043 cpu_manager.go:414] "containerMap: removing container" podUID="3b445eac-bb73-4f34-b7eb-8c7164fdfcf1" containerName="cleanup"
Oct 30 19:30:26.542760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.535606    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b445eac-bb73-4f34-b7eb-8c7164fdfcf1" containerName="cleanup"
Oct 30 19:30:26.542760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.535626    5043 cpu_manager.go:414] "containerMap: removing container" podUID="10f83fb1-b345-4e8d-bc3f-8e9ca4622d32" containerName="cleanup"
Oct 30 19:30:26.542760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.535634    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10f83fb1-b345-4e8d-bc3f-8e9ca4622d32" containerName="cleanup"
Oct 30 19:30:26.542760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.535849    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b445eac-bb73-4f34-b7eb-8c7164fdfcf1" containerName="cleanup"
Oct 30 19:30:26.542760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.535867    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b445eac-bb73-4f34-b7eb-8c7164fdfcf1" containerName="cleanup"
Oct 30 19:30:26.542760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.535880    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="10f83fb1-b345-4e8d-bc3f-8e9ca4622d32" containerName="cleanup"
Oct 30 19:30:26.542760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.535890    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="c644e491-b875-42ea-b0ec-164b96a17176" containerName="cleanup"
Oct 30 19:30:26.542760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.535907    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="10f83fb1-b345-4e8d-bc3f-8e9ca4622d32" containerName="cleanup"
Oct 30 19:30:26.542760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.535917    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="c644e491-b875-42ea-b0ec-164b96a17176" containerName="cleanup"
Oct 30 19:30:26.542760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.535925    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="10f83fb1-b345-4e8d-bc3f-8e9ca4622d32" containerName="cleanup"
Oct 30 19:30:26.542760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.535935    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="036e429a-c9b2-4c90-8a6f-6af6ee075954" containerName="cleanup"
Oct 30 19:30:26.544137 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.543931    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-235d91c2-bnzkl"
Oct 30 19:30:26.549992 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.549962    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-235d91c2-bnzkl"]
Oct 30 19:30:26.649889 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.649856    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05f835e2-2b71-40c2-ac2e-d8256c71f71c" path="/var/lib/kubelet/pods/05f835e2-2b71-40c2-ac2e-d8256c71f71c/volumes"
Oct 30 19:30:26.650570 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.650543    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10f83fb1-b345-4e8d-bc3f-8e9ca4622d32" path="/var/lib/kubelet/pods/10f83fb1-b345-4e8d-bc3f-8e9ca4622d32/volumes"
Oct 30 19:30:26.651223 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.651200    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b445eac-bb73-4f34-b7eb-8c7164fdfcf1" path="/var/lib/kubelet/pods/3b445eac-bb73-4f34-b7eb-8c7164fdfcf1/volumes"
Oct 30 19:30:26.657141 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.652099    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c644e491-b875-42ea-b0ec-164b96a17176" path="/var/lib/kubelet/pods/c644e491-b875-42ea-b0ec-164b96a17176/volumes"
Oct 30 19:30:26.657141 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.653718    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/277eda52-9140-4bc2-aacd-4b383b4ca7c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-235d91c2-bnzkl\" (UID: \"277eda52-9140-4bc2-aacd-4b383b4ca7c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-235d91c2-bnzkl"
Oct 30 19:30:26.657141 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.653882    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9khd\" (UniqueName: \"kubernetes.io/projected/277eda52-9140-4bc2-aacd-4b383b4ca7c5-kube-api-access-x9khd\") pod \"maas-api-revoke-sub-235d91c2-bnzkl\" (UID: \"277eda52-9140-4bc2-aacd-4b383b4ca7c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-235d91c2-bnzkl"
Oct 30 19:30:26.657141 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.653938    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/277eda52-9140-4bc2-aacd-4b383b4ca7c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-235d91c2-bnzkl\" (UID: \"277eda52-9140-4bc2-aacd-4b383b4ca7c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-235d91c2-bnzkl"
Oct 30 19:30:26.755192 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.755160    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/277eda52-9140-4bc2-aacd-4b383b4ca7c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-235d91c2-bnzkl\" (UID: \"277eda52-9140-4bc2-aacd-4b383b4ca7c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-235d91c2-bnzkl"
Oct 30 19:30:26.755341 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.755283    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9khd\" (UniqueName: \"kubernetes.io/projected/277eda52-9140-4bc2-aacd-4b383b4ca7c5-kube-api-access-x9khd\") pod \"maas-api-revoke-sub-235d91c2-bnzkl\" (UID: \"277eda52-9140-4bc2-aacd-4b383b4ca7c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-235d91c2-bnzkl"
Oct 30 19:30:26.755417 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.755341    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/277eda52-9140-4bc2-aacd-4b383b4ca7c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-235d91c2-bnzkl\" (UID: \"277eda52-9140-4bc2-aacd-4b383b4ca7c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-235d91c2-bnzkl"
Oct 30 19:30:26.755796 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.755773    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/277eda52-9140-4bc2-aacd-4b383b4ca7c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-235d91c2-bnzkl\" (UID: \"277eda52-9140-4bc2-aacd-4b383b4ca7c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-235d91c2-bnzkl"
Oct 30 19:30:26.757927 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.757903    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/277eda52-9140-4bc2-aacd-4b383b4ca7c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-235d91c2-bnzkl\" (UID: \"277eda52-9140-4bc2-aacd-4b383b4ca7c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-235d91c2-bnzkl"
Oct 30 19:30:26.763874 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.763854    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9khd\" (UniqueName: \"kubernetes.io/projected/277eda52-9140-4bc2-aacd-4b383b4ca7c5-kube-api-access-x9khd\") pod \"maas-api-revoke-sub-235d91c2-bnzkl\" (UID: \"277eda52-9140-4bc2-aacd-4b383b4ca7c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-235d91c2-bnzkl"
Oct 30 19:30:26.899433 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:26.899339    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-235d91c2-bnzkl"
Oct 30 19:30:27.079993 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.079960    5043 generic.go:356] "Generic (PLEG): container finished" podID="8451c454-50d0-45ef-b0c8-7b81f855e6db" containerID="573b34198874a4f72afaa00c9477052d3ac1c32dd1522437c3dba33b5b09a608" exitCode=0
Oct 30 19:30:27.080527 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.080040    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc1938a-bblth" event={"ID":"8451c454-50d0-45ef-b0c8-7b81f855e6db","Type":"ContainerDied","Data":"573b34198874a4f72afaa00c9477052d3ac1c32dd1522437c3dba33b5b09a608"}
Oct 30 19:30:27.194692 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:30:27.194663    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod277eda52_9140_4bc2_aacd_4b383b4ca7c5.slice/crio-d1af545dc86f34c15d100245cfb5a49ed900134b8260e4994b88652a7b02981a WatchSource:0}: Error finding container d1af545dc86f34c15d100245cfb5a49ed900134b8260e4994b88652a7b02981a: Status 404 returned error can't find the container with id d1af545dc86f34c15d100245cfb5a49ed900134b8260e4994b88652a7b02981a
Oct 30 19:30:27.195522 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.195497    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-235d91c2-bnzkl"]
Oct 30 19:30:27.244156 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.244126    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv"]
Oct 30 19:30:27.245242 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.245220    5043 cpu_manager.go:414] "containerMap: removing container" podUID="c644e491-b875-42ea-b0ec-164b96a17176" containerName="cleanup"
Oct 30 19:30:27.245304 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.245247    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c644e491-b875-42ea-b0ec-164b96a17176" containerName="cleanup"
Oct 30 19:30:27.245473 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.245458    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b445eac-bb73-4f34-b7eb-8c7164fdfcf1" containerName="cleanup"
Oct 30 19:30:27.245510 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.245481    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="c644e491-b875-42ea-b0ec-164b96a17176" containerName="cleanup"
Oct 30 19:30:27.251060 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.251039    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv"
Oct 30 19:30:27.263032 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.263007    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv"]
Oct 30 19:30:27.287475 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.285036    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq"]
Oct 30 19:30:27.290968 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.290945    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq"
Oct 30 19:30:27.304946 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.304919    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq"]
Oct 30 19:30:27.364315 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.364287    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-istio-podinfo\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv\" (UID: \"63688e51-3c1c-41ed-af40-4e2aa0e98c9e\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv"
Oct 30 19:30:27.364451 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.364379    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-istio-envoy\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv\" (UID: \"63688e51-3c1c-41ed-af40-4e2aa0e98c9e\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv"
Oct 30 19:30:27.364514 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.364486    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/212ce985-524a-4391-8926-63aafdc58132-istio-podinfo\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq\" (UID: \"212ce985-524a-4391-8926-63aafdc58132\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq"
Oct 30 19:30:27.364564 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.364539    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/212ce985-524a-4391-8926-63aafdc58132-credential-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq\" (UID: \"212ce985-524a-4391-8926-63aafdc58132\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq"
Oct 30 19:30:27.364608 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.364593    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/212ce985-524a-4391-8926-63aafdc58132-istio-data\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq\" (UID: \"212ce985-524a-4391-8926-63aafdc58132\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq"
Oct 30 19:30:27.364695 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.364675    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-istio-token\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv\" (UID: \"63688e51-3c1c-41ed-af40-4e2aa0e98c9e\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv"
Oct 30 19:30:27.364806 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.364790    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-istiod-ca-cert\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv\" (UID: \"63688e51-3c1c-41ed-af40-4e2aa0e98c9e\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv"
Oct 30 19:30:27.364872 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.364838    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/212ce985-524a-4391-8926-63aafdc58132-workload-certs\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq\" (UID: \"212ce985-524a-4391-8926-63aafdc58132\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq"
Oct 30 19:30:27.364935 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.364902    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/212ce985-524a-4391-8926-63aafdc58132-istio-envoy\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq\" (UID: \"212ce985-524a-4391-8926-63aafdc58132\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq"
Oct 30 19:30:27.365076 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.364952    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/212ce985-524a-4391-8926-63aafdc58132-istio-token\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq\" (UID: \"212ce985-524a-4391-8926-63aafdc58132\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq"
Oct 30 19:30:27.365076 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.364997    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-istio-data\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv\" (UID: \"63688e51-3c1c-41ed-af40-4e2aa0e98c9e\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv"
Oct 30 19:30:27.365196 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.365099    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-workload-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv\" (UID: \"63688e51-3c1c-41ed-af40-4e2aa0e98c9e\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv"
Oct 30 19:30:27.365372 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.365205    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvsks\" (UniqueName: \"kubernetes.io/projected/212ce985-524a-4391-8926-63aafdc58132-kube-api-access-tvsks\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq\" (UID: \"212ce985-524a-4391-8926-63aafdc58132\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq"
Oct 30 19:30:27.365372 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.365245    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-workload-certs\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv\" (UID: \"63688e51-3c1c-41ed-af40-4e2aa0e98c9e\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv"
Oct 30 19:30:27.365372 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.365301    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqr8x\" (UniqueName: \"kubernetes.io/projected/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-kube-api-access-fqr8x\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv\" (UID: \"63688e51-3c1c-41ed-af40-4e2aa0e98c9e\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv"
Oct 30 19:30:27.365372 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.365334    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-credential-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv\" (UID: \"63688e51-3c1c-41ed-af40-4e2aa0e98c9e\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv"
Oct 30 19:30:27.365372 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.365360    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/212ce985-524a-4391-8926-63aafdc58132-istiod-ca-cert\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq\" (UID: \"212ce985-524a-4391-8926-63aafdc58132\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq"
Oct 30 19:30:27.365563 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.365464    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/212ce985-524a-4391-8926-63aafdc58132-workload-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq\" (UID: \"212ce985-524a-4391-8926-63aafdc58132\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq"
Oct 30 19:30:27.466728 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.466699    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/212ce985-524a-4391-8926-63aafdc58132-istio-data\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq\" (UID: \"212ce985-524a-4391-8926-63aafdc58132\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq"
Oct 30 19:30:27.466924 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.466768    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-istio-token\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv\" (UID: \"63688e51-3c1c-41ed-af40-4e2aa0e98c9e\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv"
Oct 30 19:30:27.466924 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.466808    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-istiod-ca-cert\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv\" (UID: \"63688e51-3c1c-41ed-af40-4e2aa0e98c9e\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv"
Oct 30 19:30:27.466924 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.466838    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/212ce985-524a-4391-8926-63aafdc58132-workload-certs\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq\" (UID: \"212ce985-524a-4391-8926-63aafdc58132\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq"
Oct 30 19:30:27.466924 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.466866    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/212ce985-524a-4391-8926-63aafdc58132-istio-envoy\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq\" (UID: \"212ce985-524a-4391-8926-63aafdc58132\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq"
Oct 30 19:30:27.466924 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.466898    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/212ce985-524a-4391-8926-63aafdc58132-istio-token\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq\" (UID: \"212ce985-524a-4391-8926-63aafdc58132\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq"
Oct 30 19:30:27.467229 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.466935    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-istio-data\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv\" (UID: \"63688e51-3c1c-41ed-af40-4e2aa0e98c9e\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv"
Oct 30 19:30:27.467229 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.466986    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-workload-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv\" (UID: \"63688e51-3c1c-41ed-af40-4e2aa0e98c9e\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv"
Oct 30 19:30:27.467229 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.467083    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvsks\" (UniqueName: \"kubernetes.io/projected/212ce985-524a-4391-8926-63aafdc58132-kube-api-access-tvsks\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq\" (UID: \"212ce985-524a-4391-8926-63aafdc58132\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq"
Oct 30 19:30:27.467229 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.467128    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-workload-certs\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv\" (UID: \"63688e51-3c1c-41ed-af40-4e2aa0e98c9e\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv"
Oct 30 19:30:27.467229 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.467196    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqr8x\" (UniqueName: \"kubernetes.io/projected/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-kube-api-access-fqr8x\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv\" (UID: \"63688e51-3c1c-41ed-af40-4e2aa0e98c9e\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv"
Oct 30 19:30:27.467448 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.467230    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-credential-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv\" (UID: \"63688e51-3c1c-41ed-af40-4e2aa0e98c9e\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv"
Oct 30 19:30:27.467448 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.467303    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/212ce985-524a-4391-8926-63aafdc58132-istiod-ca-cert\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq\" (UID: \"212ce985-524a-4391-8926-63aafdc58132\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq"
Oct 30 19:30:27.467448 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.467338    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/212ce985-524a-4391-8926-63aafdc58132-workload-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq\" (UID: \"212ce985-524a-4391-8926-63aafdc58132\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq"
Oct 30 19:30:27.467448 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.467378    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-istio-podinfo\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv\" (UID: \"63688e51-3c1c-41ed-af40-4e2aa0e98c9e\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv"
Oct 30 19:30:27.467586 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.467458    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-istio-envoy\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv\" (UID: \"63688e51-3c1c-41ed-af40-4e2aa0e98c9e\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv"
Oct 30 19:30:27.467586 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.467473    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-workload-certs\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv\" (UID: \"63688e51-3c1c-41ed-af40-4e2aa0e98c9e\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv"
Oct 30 19:30:27.467586 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.467514    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-istiod-ca-cert\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv\" (UID: \"63688e51-3c1c-41ed-af40-4e2aa0e98c9e\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv"
Oct 30 19:30:27.467586 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.467247    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/212ce985-524a-4391-8926-63aafdc58132-workload-certs\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq\" (UID: \"212ce985-524a-4391-8926-63aafdc58132\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq"
Oct 30 19:30:27.467586 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.467540    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/212ce985-524a-4391-8926-63aafdc58132-istio-podinfo\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq\" (UID: \"212ce985-524a-4391-8926-63aafdc58132\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq"
Oct 30 19:30:27.467855 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.467603    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-workload-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv\" (UID: \"63688e51-3c1c-41ed-af40-4e2aa0e98c9e\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv"
Oct 30 19:30:27.467855 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.467610    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/212ce985-524a-4391-8926-63aafdc58132-credential-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq\" (UID: \"212ce985-524a-4391-8926-63aafdc58132\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq"
Oct 30 19:30:27.467960 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.467876    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/212ce985-524a-4391-8926-63aafdc58132-credential-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq\" (UID: \"212ce985-524a-4391-8926-63aafdc58132\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq"
Oct 30 19:30:27.468839 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.468814    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-credential-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv\" (UID: \"63688e51-3c1c-41ed-af40-4e2aa0e98c9e\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv"
Oct 30 19:30:27.469267 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.469244    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/212ce985-524a-4391-8926-63aafdc58132-workload-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq\" (UID: \"212ce985-524a-4391-8926-63aafdc58132\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq"
Oct 30 19:30:27.469680 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.469654    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/212ce985-524a-4391-8926-63aafdc58132-istio-envoy\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq\" (UID: \"212ce985-524a-4391-8926-63aafdc58132\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq"
Oct 30 19:30:27.469892 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.469843    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/212ce985-524a-4391-8926-63aafdc58132-istiod-ca-cert\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq\" (UID: \"212ce985-524a-4391-8926-63aafdc58132\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq"
Oct 30 19:30:27.469982 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.469904    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-istio-envoy\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv\" (UID: \"63688e51-3c1c-41ed-af40-4e2aa0e98c9e\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv"
Oct 30 19:30:27.470105 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.470085    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/212ce985-524a-4391-8926-63aafdc58132-istio-data\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq\" (UID: \"212ce985-524a-4391-8926-63aafdc58132\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq"
Oct 30 19:30:27.470269 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.470253    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-istio-data\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv\" (UID: \"63688e51-3c1c-41ed-af40-4e2aa0e98c9e\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv"
Oct 30 19:30:27.470458 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.470441    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/212ce985-524a-4391-8926-63aafdc58132-istio-podinfo\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq\" (UID: \"212ce985-524a-4391-8926-63aafdc58132\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq"
Oct 30 19:30:27.470665 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.470649    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-istio-podinfo\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv\" (UID: \"63688e51-3c1c-41ed-af40-4e2aa0e98c9e\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv"
Oct 30 19:30:27.475158 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.475131    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-istio-token\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv\" (UID: \"63688e51-3c1c-41ed-af40-4e2aa0e98c9e\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv"
Oct 30 19:30:27.475482 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.475462    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/212ce985-524a-4391-8926-63aafdc58132-istio-token\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq\" (UID: \"212ce985-524a-4391-8926-63aafdc58132\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq"
Oct 30 19:30:27.476175 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.476152    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvsks\" (UniqueName: \"kubernetes.io/projected/212ce985-524a-4391-8926-63aafdc58132-kube-api-access-tvsks\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq\" (UID: \"212ce985-524a-4391-8926-63aafdc58132\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq"
Oct 30 19:30:27.480101 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.480079    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqr8x\" (UniqueName: \"kubernetes.io/projected/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-kube-api-access-fqr8x\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv\" (UID: \"63688e51-3c1c-41ed-af40-4e2aa0e98c9e\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv"
Oct 30 19:30:27.566364 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.566334    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm"]
Oct 30 19:30:27.742465 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.577402    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv"
Oct 30 19:30:27.742465 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.617692    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq"
Oct 30 19:30:27.742465 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.676544    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm"
Oct 30 19:30:27.742465 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.716190    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm"]
Oct 30 19:30:27.808853 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.808666    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c6df78-8285-46c7-b5bd-896a0350c1de-workload-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm\" (UID: \"03c6df78-8285-46c7-b5bd-896a0350c1de\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm"
Oct 30 19:30:27.809281 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.809254    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c6df78-8285-46c7-b5bd-896a0350c1de-credential-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm\" (UID: \"03c6df78-8285-46c7-b5bd-896a0350c1de\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm"
Oct 30 19:30:27.811178 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.811013    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh"]
Oct 30 19:30:27.902143 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.901207    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh"]
Oct 30 19:30:27.902143 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.901320    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh"
Oct 30 19:30:27.942982 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.932523    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03c6df78-8285-46c7-b5bd-896a0350c1de-istio-envoy\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm\" (UID: \"03c6df78-8285-46c7-b5bd-896a0350c1de\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm"
Oct 30 19:30:27.942982 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.932592    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnvzf\" (UniqueName: \"kubernetes.io/projected/03c6df78-8285-46c7-b5bd-896a0350c1de-kube-api-access-qnvzf\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm\" (UID: \"03c6df78-8285-46c7-b5bd-896a0350c1de\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm"
Oct 30 19:30:27.942982 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.932657    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03c6df78-8285-46c7-b5bd-896a0350c1de-workload-certs\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm\" (UID: \"03c6df78-8285-46c7-b5bd-896a0350c1de\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm"
Oct 30 19:30:27.942982 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.939499    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03c6df78-8285-46c7-b5bd-896a0350c1de-istio-podinfo\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm\" (UID: \"03c6df78-8285-46c7-b5bd-896a0350c1de\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm"
Oct 30 19:30:27.942982 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.939597    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03c6df78-8285-46c7-b5bd-896a0350c1de-istio-token\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm\" (UID: \"03c6df78-8285-46c7-b5bd-896a0350c1de\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm"
Oct 30 19:30:27.942982 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.939631    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03c6df78-8285-46c7-b5bd-896a0350c1de-istiod-ca-cert\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm\" (UID: \"03c6df78-8285-46c7-b5bd-896a0350c1de\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm"
Oct 30 19:30:27.942982 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.939656    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03c6df78-8285-46c7-b5bd-896a0350c1de-istio-data\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm\" (UID: \"03c6df78-8285-46c7-b5bd-896a0350c1de\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm"
Oct 30 19:30:27.942982 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.939689    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c6df78-8285-46c7-b5bd-896a0350c1de-workload-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm\" (UID: \"03c6df78-8285-46c7-b5bd-896a0350c1de\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm"
Oct 30 19:30:27.942982 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.939715    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c6df78-8285-46c7-b5bd-896a0350c1de-credential-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm\" (UID: \"03c6df78-8285-46c7-b5bd-896a0350c1de\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm"
Oct 30 19:30:27.942982 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.940949    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c6df78-8285-46c7-b5bd-896a0350c1de-workload-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm\" (UID: \"03c6df78-8285-46c7-b5bd-896a0350c1de\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm"
Oct 30 19:30:27.942982 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:27.941222    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c6df78-8285-46c7-b5bd-896a0350c1de-credential-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm\" (UID: \"03c6df78-8285-46c7-b5bd-896a0350c1de\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm"
Oct 30 19:30:28.022220 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.019935    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dff807b2-dtrbk"]
Oct 30 19:30:28.026015 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.025991    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff807b2-dtrbk"
Oct 30 19:30:28.034721 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.034674    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dff807b2-dtrbk"]
Oct 30 19:30:28.043149 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.043115    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa81e41-151e-47b9-8a09-f4cf20a52dad-workload-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh\" (UID: \"0aa81e41-151e-47b9-8a09-f4cf20a52dad\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh"
Oct 30 19:30:28.043302 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.043289    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mn8r\" (UniqueName: \"kubernetes.io/projected/0aa81e41-151e-47b9-8a09-f4cf20a52dad-kube-api-access-7mn8r\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh\" (UID: \"0aa81e41-151e-47b9-8a09-f4cf20a52dad\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh"
Oct 30 19:30:28.043375 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.043364    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03c6df78-8285-46c7-b5bd-896a0350c1de-istio-podinfo\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm\" (UID: \"03c6df78-8285-46c7-b5bd-896a0350c1de\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm"
Oct 30 19:30:28.043499 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.043488    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03c6df78-8285-46c7-b5bd-896a0350c1de-istio-token\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm\" (UID: \"03c6df78-8285-46c7-b5bd-896a0350c1de\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm"
Oct 30 19:30:28.043572 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.043562    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03c6df78-8285-46c7-b5bd-896a0350c1de-istiod-ca-cert\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm\" (UID: \"03c6df78-8285-46c7-b5bd-896a0350c1de\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm"
Oct 30 19:30:28.043628 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.043620    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03c6df78-8285-46c7-b5bd-896a0350c1de-istio-data\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm\" (UID: \"03c6df78-8285-46c7-b5bd-896a0350c1de\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm"
Oct 30 19:30:28.043729 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.043717    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa81e41-151e-47b9-8a09-f4cf20a52dad-credential-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh\" (UID: \"0aa81e41-151e-47b9-8a09-f4cf20a52dad\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh"
Oct 30 19:30:28.043929 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.043915    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aa81e41-151e-47b9-8a09-f4cf20a52dad-istio-data\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh\" (UID: \"0aa81e41-151e-47b9-8a09-f4cf20a52dad\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh"
Oct 30 19:30:28.044057 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.044046    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aa81e41-151e-47b9-8a09-f4cf20a52dad-istio-podinfo\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh\" (UID: \"0aa81e41-151e-47b9-8a09-f4cf20a52dad\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh"
Oct 30 19:30:28.044226 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.044216    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aa81e41-151e-47b9-8a09-f4cf20a52dad-istiod-ca-cert\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh\" (UID: \"0aa81e41-151e-47b9-8a09-f4cf20a52dad\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh"
Oct 30 19:30:28.044896 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.044820    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0aa81e41-151e-47b9-8a09-f4cf20a52dad-istio-token\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh\" (UID: \"0aa81e41-151e-47b9-8a09-f4cf20a52dad\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh"
Oct 30 19:30:28.044896 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.044874    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03c6df78-8285-46c7-b5bd-896a0350c1de-istio-envoy\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm\" (UID: \"03c6df78-8285-46c7-b5bd-896a0350c1de\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm"
Oct 30 19:30:28.045101 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.044608    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03c6df78-8285-46c7-b5bd-896a0350c1de-istio-data\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm\" (UID: \"03c6df78-8285-46c7-b5bd-896a0350c1de\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm"
Oct 30 19:30:28.045470 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.045408    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03c6df78-8285-46c7-b5bd-896a0350c1de-istiod-ca-cert\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm\" (UID: \"03c6df78-8285-46c7-b5bd-896a0350c1de\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm"
Oct 30 19:30:28.046017 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.045938    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnvzf\" (UniqueName: \"kubernetes.io/projected/03c6df78-8285-46c7-b5bd-896a0350c1de-kube-api-access-qnvzf\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm\" (UID: \"03c6df78-8285-46c7-b5bd-896a0350c1de\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm"
Oct 30 19:30:28.046256 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.046152    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03c6df78-8285-46c7-b5bd-896a0350c1de-workload-certs\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm\" (UID: \"03c6df78-8285-46c7-b5bd-896a0350c1de\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm"
Oct 30 19:30:28.046256 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.046223    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aa81e41-151e-47b9-8a09-f4cf20a52dad-workload-certs\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh\" (UID: \"0aa81e41-151e-47b9-8a09-f4cf20a52dad\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh"
Oct 30 19:30:28.047652 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.046509    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aa81e41-151e-47b9-8a09-f4cf20a52dad-istio-envoy\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh\" (UID: \"0aa81e41-151e-47b9-8a09-f4cf20a52dad\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh"
Oct 30 19:30:28.048862 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.048837    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03c6df78-8285-46c7-b5bd-896a0350c1de-istio-envoy\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm\" (UID: \"03c6df78-8285-46c7-b5bd-896a0350c1de\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm"
Oct 30 19:30:28.048963 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.048854    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03c6df78-8285-46c7-b5bd-896a0350c1de-workload-certs\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm\" (UID: \"03c6df78-8285-46c7-b5bd-896a0350c1de\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm"
Oct 30 19:30:28.135797 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.135000    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03c6df78-8285-46c7-b5bd-896a0350c1de-istio-token\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm\" (UID: \"03c6df78-8285-46c7-b5bd-896a0350c1de\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm"
Oct 30 19:30:28.151757 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.149641    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aa81e41-151e-47b9-8a09-f4cf20a52dad-workload-certs\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh\" (UID: \"0aa81e41-151e-47b9-8a09-f4cf20a52dad\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh"
Oct 30 19:30:28.151757 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.149713    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aa81e41-151e-47b9-8a09-f4cf20a52dad-istio-envoy\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh\" (UID: \"0aa81e41-151e-47b9-8a09-f4cf20a52dad\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh"
Oct 30 19:30:28.151757 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.149780    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa81e41-151e-47b9-8a09-f4cf20a52dad-workload-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh\" (UID: \"0aa81e41-151e-47b9-8a09-f4cf20a52dad\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh"
Oct 30 19:30:28.151757 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.149831    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mn8r\" (UniqueName: \"kubernetes.io/projected/0aa81e41-151e-47b9-8a09-f4cf20a52dad-kube-api-access-7mn8r\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh\" (UID: \"0aa81e41-151e-47b9-8a09-f4cf20a52dad\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh"
Oct 30 19:30:28.151757 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.149991    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa81e41-151e-47b9-8a09-f4cf20a52dad-credential-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh\" (UID: \"0aa81e41-151e-47b9-8a09-f4cf20a52dad\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh"
Oct 30 19:30:28.151757 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.150030    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aa81e41-151e-47b9-8a09-f4cf20a52dad-istio-data\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh\" (UID: \"0aa81e41-151e-47b9-8a09-f4cf20a52dad\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh"
Oct 30 19:30:28.151757 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.150063    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aa81e41-151e-47b9-8a09-f4cf20a52dad-istio-podinfo\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh\" (UID: \"0aa81e41-151e-47b9-8a09-f4cf20a52dad\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh"
Oct 30 19:30:28.151757 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.150118    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f42c708e-e9b6-4cd9-9cd4-090a55157421-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dff807b2-dtrbk\" (UID: \"f42c708e-e9b6-4cd9-9cd4-090a55157421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff807b2-dtrbk"
Oct 30 19:30:28.151757 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.150221    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aa81e41-151e-47b9-8a09-f4cf20a52dad-istiod-ca-cert\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh\" (UID: \"0aa81e41-151e-47b9-8a09-f4cf20a52dad\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh"
Oct 30 19:30:28.151757 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.150254    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f42c708e-e9b6-4cd9-9cd4-090a55157421-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dff807b2-dtrbk\" (UID: \"f42c708e-e9b6-4cd9-9cd4-090a55157421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff807b2-dtrbk"
Oct 30 19:30:28.151757 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.150279    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0aa81e41-151e-47b9-8a09-f4cf20a52dad-istio-token\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh\" (UID: \"0aa81e41-151e-47b9-8a09-f4cf20a52dad\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh"
Oct 30 19:30:28.151757 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.150336    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7qzx\" (UniqueName: \"kubernetes.io/projected/f42c708e-e9b6-4cd9-9cd4-090a55157421-kube-api-access-k7qzx\") pod \"maas-api-revoke-sub-dff807b2-dtrbk\" (UID: \"f42c708e-e9b6-4cd9-9cd4-090a55157421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff807b2-dtrbk"
Oct 30 19:30:28.259793 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.170995    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aa81e41-151e-47b9-8a09-f4cf20a52dad-workload-certs\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh\" (UID: \"0aa81e41-151e-47b9-8a09-f4cf20a52dad\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh"
Oct 30 19:30:28.259793 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.180461    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnvzf\" (UniqueName: \"kubernetes.io/projected/03c6df78-8285-46c7-b5bd-896a0350c1de-kube-api-access-qnvzf\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm\" (UID: \"03c6df78-8285-46c7-b5bd-896a0350c1de\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm"
Oct 30 19:30:28.259793 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.188125    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa81e41-151e-47b9-8a09-f4cf20a52dad-workload-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh\" (UID: \"0aa81e41-151e-47b9-8a09-f4cf20a52dad\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh"
Oct 30 19:30:28.259793 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.188375    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa81e41-151e-47b9-8a09-f4cf20a52dad-credential-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh\" (UID: \"0aa81e41-151e-47b9-8a09-f4cf20a52dad\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh"
Oct 30 19:30:28.259793 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.194129    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aa81e41-151e-47b9-8a09-f4cf20a52dad-istiod-ca-cert\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh\" (UID: \"0aa81e41-151e-47b9-8a09-f4cf20a52dad\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh"
Oct 30 19:30:28.259793 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.215119    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aa81e41-151e-47b9-8a09-f4cf20a52dad-istio-data\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh\" (UID: \"0aa81e41-151e-47b9-8a09-f4cf20a52dad\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh"
Oct 30 19:30:28.259793 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.239449    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mn8r\" (UniqueName: \"kubernetes.io/projected/0aa81e41-151e-47b9-8a09-f4cf20a52dad-kube-api-access-7mn8r\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh\" (UID: \"0aa81e41-151e-47b9-8a09-f4cf20a52dad\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh"
Oct 30 19:30:28.285161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.270324    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f42c708e-e9b6-4cd9-9cd4-090a55157421-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dff807b2-dtrbk\" (UID: \"f42c708e-e9b6-4cd9-9cd4-090a55157421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff807b2-dtrbk"
Oct 30 19:30:28.285161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.270464    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f42c708e-e9b6-4cd9-9cd4-090a55157421-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dff807b2-dtrbk\" (UID: \"f42c708e-e9b6-4cd9-9cd4-090a55157421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff807b2-dtrbk"
Oct 30 19:30:28.285161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.270524    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7qzx\" (UniqueName: \"kubernetes.io/projected/f42c708e-e9b6-4cd9-9cd4-090a55157421-kube-api-access-k7qzx\") pod \"maas-api-revoke-sub-dff807b2-dtrbk\" (UID: \"f42c708e-e9b6-4cd9-9cd4-090a55157421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff807b2-dtrbk"
Oct 30 19:30:28.285161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.271959    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f42c708e-e9b6-4cd9-9cd4-090a55157421-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dff807b2-dtrbk\" (UID: \"f42c708e-e9b6-4cd9-9cd4-090a55157421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff807b2-dtrbk"
Oct 30 19:30:28.285161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.274300    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0aa81e41-151e-47b9-8a09-f4cf20a52dad-istio-token\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh\" (UID: \"0aa81e41-151e-47b9-8a09-f4cf20a52dad\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh"
Oct 30 19:30:28.340345 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.340307    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-235d91c2-bnzkl" event={"ID":"277eda52-9140-4bc2-aacd-4b383b4ca7c5","Type":"ContainerStarted","Data":"cd805e4c2201c08dadc1a5ca6678f7f3ada40453b59eeac2a454cf3d455a00cb"}
Oct 30 19:30:28.345769 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.340599    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-235d91c2-bnzkl" event={"ID":"277eda52-9140-4bc2-aacd-4b383b4ca7c5","Type":"ContainerStarted","Data":"d1af545dc86f34c15d100245cfb5a49ed900134b8260e4994b88652a7b02981a"}
Oct 30 19:30:28.472759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.472241    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03c6df78-8285-46c7-b5bd-896a0350c1de-istio-podinfo\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm\" (UID: \"03c6df78-8285-46c7-b5bd-896a0350c1de\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm"
Oct 30 19:30:28.597818 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.586562    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-235d91c2-bnzkl" podStartSLOduration=2.586540749 podStartE2EDuration="2.586540749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:28.407449196 +0000 UTC m=+847.918509161" watchObservedRunningTime="2026-10-30 19:30:28.586540749 +0000 UTC m=+848.097600715"
Oct 30 19:30:28.666762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.602645    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7qzx\" (UniqueName: \"kubernetes.io/projected/f42c708e-e9b6-4cd9-9cd4-090a55157421-kube-api-access-k7qzx\") pod \"maas-api-revoke-sub-dff807b2-dtrbk\" (UID: \"f42c708e-e9b6-4cd9-9cd4-090a55157421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff807b2-dtrbk"
Oct 30 19:30:28.671244 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.603190    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm"
Oct 30 19:30:28.671244 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.656427    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aa81e41-151e-47b9-8a09-f4cf20a52dad-istio-podinfo\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh\" (UID: \"0aa81e41-151e-47b9-8a09-f4cf20a52dad\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh"
Oct 30 19:30:28.672772 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.672726    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aa81e41-151e-47b9-8a09-f4cf20a52dad-istio-envoy\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh\" (UID: \"0aa81e41-151e-47b9-8a09-f4cf20a52dad\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh"
Oct 30 19:30:28.684083 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.683942    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh"
Oct 30 19:30:28.757774 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.747499    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f42c708e-e9b6-4cd9-9cd4-090a55157421-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dff807b2-dtrbk\" (UID: \"f42c708e-e9b6-4cd9-9cd4-090a55157421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff807b2-dtrbk"
Oct 30 19:30:28.878331 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.878298    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff807b2-dtrbk"
Oct 30 19:30:28.959865 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:28.959835    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv"]
Oct 30 19:30:29.107781 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:29.102915    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:29.134870 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:29.126806    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:29.134870 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:29.128199    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:30:29.216244 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:29.216202    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:29.297149 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:30:29.277886    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod212ce985_524a_4391_8926_63aafdc58132.slice/crio-32348090f808fc56f6bedb793032dccfb75e599640dd82acdaaea6354824dd98 WatchSource:0}: Error finding container 32348090f808fc56f6bedb793032dccfb75e599640dd82acdaaea6354824dd98: Status 404 returned error can't find the container with id 32348090f808fc56f6bedb793032dccfb75e599640dd82acdaaea6354824dd98
Oct 30 19:30:29.437107 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:29.424848    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:30:29.437107 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:29.424946    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:30:29.444004 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:29.443973    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq"]
Oct 30 19:30:29.490995 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:29.490948    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:30:29.510763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:29.510403    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:29.587878 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:29.587827    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:30:29.591761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:29.588445    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:30:29.662765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:29.622801    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:29.843772 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:29.825863    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:29.849097 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:29.849057    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": context deadline exceeded"
Oct 30 19:30:29.966359 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:29.966307    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:30.057207 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:30.057115    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:30:30.143476 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:30.095529    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:30.143476 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:30.122345    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:30:30.143476 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:30.122425    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:30:30.143476 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:30.138864    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:30.143476 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:30.139262    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:30:30.245171 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:30.230026    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:30.245171 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:30.236918    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:30.245171 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:30.237762    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": context deadline exceeded"
Oct 30 19:30:30.498773 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:30.498568    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:30:30.550807 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:30.513115    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:30:30.550807 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:30.514666    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:30.550807 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:30.539470    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:30:30.587315 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:30.562304    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq" event={"ID":"212ce985-524a-4391-8926-63aafdc58132","Type":"ContainerStarted","Data":"32348090f808fc56f6bedb793032dccfb75e599640dd82acdaaea6354824dd98"}
Oct 30 19:30:30.588255 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:30.588215    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:30.590046 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:30.590023    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:30:30.705820 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:30.620661    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:30.722763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:30.718863    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:30.903674 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:30.890876    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:30.912421 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:30.912393    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:30:30.912641 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:30.912618    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv" event={"ID":"63688e51-3c1c-41ed-af40-4e2aa0e98c9e","Type":"ContainerStarted","Data":"dad9c7a20312978b5d423f67c0490a11e154b8bc971bac7371d036b5dc07e58d"}
Oct 30 19:30:30.912967 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:30.912948    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"
Oct 30 19:30:30.913130 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:30.913117    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:30:30.943204 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:30.942876    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc1938a-bblth"
Oct 30 19:30:31.046303 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:31.006867    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:31.052873 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:31.052836    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:30:31.539005 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:31.532869    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:31.539005 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:31.534650    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:31.697867 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:31.696864    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:31.755783 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:31.751195    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:31.903806 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:31.831334    5043 generic.go:356] "Generic (PLEG): container finished" podID="277eda52-9140-4bc2-aacd-4b383b4ca7c5" containerID="cd805e4c2201c08dadc1a5ca6678f7f3ada40453b59eeac2a454cf3d455a00cb" exitCode=0
Oct 30 19:30:31.903806 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:31.832145    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:31.903806 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:31.879345    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:31.983761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:31.968818    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:32.036072 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:31.973183    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:30:32.036072 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:31.973362    5043 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:30:32.036072 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:32.020281    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr" podUID="6251d2e1-9d28-4bb8-a667-f415d0f79df0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:32.071761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:32.071602    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-6dg26" podUID="692c558e-cc23-4533-b6ad-0f982a82c187" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:32.205078 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:32.205037    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:33.237903 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:33.215989    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:30:33.237903 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:33.216295    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:30:33.237903 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:33.216448    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:30:33.269131 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:33.269089    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:30:33.485758 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:33.463985    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8451c454-50d0-45ef-b0c8-7b81f855e6db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8451c454-50d0-45ef-b0c8-7b81f855e6db-maas-api-openshift-ca\") pod \"8451c454-50d0-45ef-b0c8-7b81f855e6db\" (UID: \"8451c454-50d0-45ef-b0c8-7b81f855e6db\") "
Oct 30 19:30:33.486086 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:33.486062    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8451c454-50d0-45ef-b0c8-7b81f855e6db-kube-api-access-rrs5n\" (UniqueName: \"kubernetes.io/projected/8451c454-50d0-45ef-b0c8-7b81f855e6db-kube-api-access-rrs5n\") pod \"8451c454-50d0-45ef-b0c8-7b81f855e6db\" (UID: \"8451c454-50d0-45ef-b0c8-7b81f855e6db\") "
Oct 30 19:30:33.486361 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:33.486331    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8451c454-50d0-45ef-b0c8-7b81f855e6db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8451c454-50d0-45ef-b0c8-7b81f855e6db-maas-api-xks-ca\") pod \"8451c454-50d0-45ef-b0c8-7b81f855e6db\" (UID: \"8451c454-50d0-45ef-b0c8-7b81f855e6db\") "
Oct 30 19:30:33.691761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:33.661481    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:30:34.680164 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:33.680380    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8451c454-50d0-45ef-b0c8-7b81f855e6db-maas-api-openshift-ca" pod "8451c454-50d0-45ef-b0c8-7b81f855e6db" (UID: "8451c454-50d0-45ef-b0c8-7b81f855e6db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:34.680723 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:33.725356    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8451c454-50d0-45ef-b0c8-7b81f855e6db-kube-api-access-rrs5n" pod "8451c454-50d0-45ef-b0c8-7b81f855e6db" (UID: "8451c454-50d0-45ef-b0c8-7b81f855e6db"). InnerVolumeSpecName "kube-api-access-rrs5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:34.737788 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:34.737755    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"
Oct 30 19:30:34.788063 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:34.786786    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8451c454-50d0-45ef-b0c8-7b81f855e6db-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:34.788063 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:34.786816    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrs5n\" (UniqueName: \"kubernetes.io/projected/8451c454-50d0-45ef-b0c8-7b81f855e6db-kube-api-access-rrs5n\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:34.804640 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:34.804603    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:34.822639 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:34.822611    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8451c454-50d0-45ef-b0c8-7b81f855e6db-maas-api-xks-ca" pod "8451c454-50d0-45ef-b0c8-7b81f855e6db" (UID: "8451c454-50d0-45ef-b0c8-7b81f855e6db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:30:34.828969 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:34.818385    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:34.838340 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:30:34.838310    5043 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf42c708e_e9b6_4cd9_9cd4_090a55157421.slice/crio-53531fec56ee6b2afc6b17c756c6e2f122d082ab22addd149b6601827dcbb267\": RecentStats: unable to find data in memory cache]"
Oct 30 19:30:34.889183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:34.883933    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc1938a-bblth"
Oct 30 19:30:34.903805 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:34.896233    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:30:34.903805 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:34.896322    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:30:34.903805 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:34.896363    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:30:34.903805 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:34.896802    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8451c454-50d0-45ef-b0c8-7b81f855e6db-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:34.922834 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:34.922807    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-235d91c2-bnzkl" event={"ID":"277eda52-9140-4bc2-aacd-4b383b4ca7c5","Type":"ContainerDied","Data":"cd805e4c2201c08dadc1a5ca6678f7f3ada40453b59eeac2a454cf3d455a00cb"}
Oct 30 19:30:34.923034 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:34.923021    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:30:34.923129 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:34.923117    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc1938a-bblth" event={"ID":"8451c454-50d0-45ef-b0c8-7b81f855e6db","Type":"ContainerDied","Data":"e90521784b19651c2e49be7a3d564144ecd2c4fc87cfbd60cf6489c84dc7590c"}
Oct 30 19:30:34.923224 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:34.923213    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e90521784b19651c2e49be7a3d564144ecd2c4fc87cfbd60cf6489c84dc7590c"
Oct 30 19:30:34.923320 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:34.923311    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:30:34.923431 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:34.923422    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:30:34.923525 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:34.923517    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:30:34.923618 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:34.923610    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:30:34.929630 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:34.923724    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"
Oct 30 19:30:34.930320 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:34.930250    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm" event={"ID":"03c6df78-8285-46c7-b5bd-896a0350c1de","Type":"ContainerStarted","Data":"4fe589a72875ebe06416f6c51d18b8698e3a515be49eca928beaac94b5ab821a"}
Oct 30 19:30:34.930542 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:34.930525    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:30:34.930693 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:34.930679    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"
Oct 30 19:30:34.930871 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:34.930858    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:30:34.931037 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:34.931022    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm"]
Oct 30 19:30:34.943626 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:34.938010    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dff807b2-dtrbk"]
Oct 30 19:30:35.128710 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:35.128686    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh"]
Oct 30 19:30:35.531876 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:35.531797    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv" podStartSLOduration=8.531778831 podStartE2EDuration="8.531778831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:35.228119387 +0000 UTC m=+854.739179354" watchObservedRunningTime="2026-10-30 19:30:35.531778831 +0000 UTC m=+855.042838795"
Oct 30 19:30:35.608555 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:35.608160    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv"
Oct 30 19:30:35.609191 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:35.608878    5043 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 19:30:35.609191 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:35.608930    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv" podUID="63688e51-3c1c-41ed-af40-4e2aa0e98c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 19:30:35.653830 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:35.634248    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq" podStartSLOduration=8.634228545 podStartE2EDuration="8.634228545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:35.246461787 +0000 UTC m=+854.757521754" watchObservedRunningTime="2026-10-30 19:30:35.634228545 +0000 UTC m=+855.145288514"
Oct 30 19:30:35.653830 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:35.652945    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq"
Oct 30 19:30:35.653830 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:35.653054    5043 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Oct 30 19:30:35.653830 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:35.653092    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq" podUID="212ce985-524a-4391-8926-63aafdc58132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Oct 30 19:30:36.036666 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:36.036621    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80a9dfa1-mn2sk"]
Oct 30 19:30:36.097189 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:36.097157    5043 cpu_manager.go:414] "containerMap: removing container" podUID="8451c454-50d0-45ef-b0c8-7b81f855e6db" containerName="revoke-keys"
Oct 30 19:30:36.097439 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:36.097409    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8451c454-50d0-45ef-b0c8-7b81f855e6db" containerName="revoke-keys"
Oct 30 19:30:36.123331 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:36.123300    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="8451c454-50d0-45ef-b0c8-7b81f855e6db" containerName="revoke-keys"
Oct 30 19:30:36.245023 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:36.244970    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80a9dfa1-mn2sk"]
Oct 30 19:30:36.245319 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:36.245295    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh" event={"ID":"0aa81e41-151e-47b9-8a09-f4cf20a52dad","Type":"ContainerStarted","Data":"4a572a7d72678e26f0fb808f9c6c84dbc1db2049e9f5185d0a0c6faf228a31aa"}
Oct 30 19:30:36.290593 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:36.290544    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh" event={"ID":"0aa81e41-151e-47b9-8a09-f4cf20a52dad","Type":"ContainerStarted","Data":"88eed1fe5c7124798357ddbb5a205136528dccd701fc104ea2c5fea51d3c138b"}
Oct 30 19:30:36.290811 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:36.290793    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm" event={"ID":"03c6df78-8285-46c7-b5bd-896a0350c1de","Type":"ContainerStarted","Data":"92e838cda70530f6ebca3abe2d1d3da143d9c63d8038229c8fedefa8609345d7"}
Oct 30 19:30:36.290982 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:36.248187    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh" podStartSLOduration=9.248170717 podStartE2EDuration="9.248170717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:36.171952032 +0000 UTC m=+855.683012000" watchObservedRunningTime="2026-10-30 19:30:36.248170717 +0000 UTC m=+855.759230684"
Oct 30 19:30:36.291837 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:36.245010    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a9dfa1-mn2sk"
Oct 30 19:30:36.341761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:36.308520    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sk69\" (UniqueName: \"kubernetes.io/projected/6095ad00-f596-438a-96c7-80757d6c8284-kube-api-access-8sk69\") pod \"maas-api-revoke-sub-80a9dfa1-mn2sk\" (UID: \"6095ad00-f596-438a-96c7-80757d6c8284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a9dfa1-mn2sk"
Oct 30 19:30:36.341761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:36.308783    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6095ad00-f596-438a-96c7-80757d6c8284-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80a9dfa1-mn2sk\" (UID: \"6095ad00-f596-438a-96c7-80757d6c8284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a9dfa1-mn2sk"
Oct 30 19:30:36.341761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:36.308902    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6095ad00-f596-438a-96c7-80757d6c8284-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80a9dfa1-mn2sk\" (UID: \"6095ad00-f596-438a-96c7-80757d6c8284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a9dfa1-mn2sk"
Oct 30 19:30:36.545820 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:36.474487    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sk69\" (UniqueName: \"kubernetes.io/projected/6095ad00-f596-438a-96c7-80757d6c8284-kube-api-access-8sk69\") pod \"maas-api-revoke-sub-80a9dfa1-mn2sk\" (UID: \"6095ad00-f596-438a-96c7-80757d6c8284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a9dfa1-mn2sk"
Oct 30 19:30:36.545820 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:36.474654    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6095ad00-f596-438a-96c7-80757d6c8284-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80a9dfa1-mn2sk\" (UID: \"6095ad00-f596-438a-96c7-80757d6c8284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a9dfa1-mn2sk"
Oct 30 19:30:36.651774 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:36.550125    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq" event={"ID":"212ce985-524a-4391-8926-63aafdc58132","Type":"ContainerStarted","Data":"35ffcde418b19f88d351e011ed9b566bfe3f6111991ee359720fd4947da07efb"}
Oct 30 19:30:36.651774 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:36.580787    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv" event={"ID":"63688e51-3c1c-41ed-af40-4e2aa0e98c9e","Type":"ContainerStarted","Data":"e548144cfc3d45364a127a8f8247633b91b481e7ab36993163e27edda2c15d04"}
Oct 30 19:30:36.787762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:36.765608    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6095ad00-f596-438a-96c7-80757d6c8284-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80a9dfa1-mn2sk\" (UID: \"6095ad00-f596-438a-96c7-80757d6c8284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a9dfa1-mn2sk"
Oct 30 19:30:36.787762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:36.766805    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm" podStartSLOduration=9.766787026 podStartE2EDuration="9.766787026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:36.228369963 +0000 UTC m=+855.739429932" watchObservedRunningTime="2026-10-30 19:30:36.766787026 +0000 UTC m=+856.277846991"
Oct 30 19:30:36.791081 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:36.791038    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6095ad00-f596-438a-96c7-80757d6c8284-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80a9dfa1-mn2sk\" (UID: \"6095ad00-f596-438a-96c7-80757d6c8284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a9dfa1-mn2sk"
Oct 30 19:30:36.795500 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:36.795469    5043 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 30 19:30:36.795695 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:36.795669    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh" podUID="0aa81e41-151e-47b9-8a09-f4cf20a52dad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 30 19:30:36.819811 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:36.802572    5043 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 19:30:36.940760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:36.833890    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh"
Oct 30 19:30:36.940760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:36.916018    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff807b2-dtrbk" event={"ID":"f42c708e-e9b6-4cd9-9cd4-090a55157421","Type":"ContainerStarted","Data":"30de05097ec1b9339ebde00cd70dbf95e988c97f28557320bbdf2482d1e87ac5"}
Oct 30 19:30:36.940760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:36.916058    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff807b2-dtrbk" event={"ID":"f42c708e-e9b6-4cd9-9cd4-090a55157421","Type":"ContainerStarted","Data":"53531fec56ee6b2afc6b17c756c6e2f122d082ab22addd149b6601827dcbb267"}
Oct 30 19:30:36.985149 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:36.985096    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv" podUID="63688e51-3c1c-41ed-af40-4e2aa0e98c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 19:30:37.021813 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:37.018690    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6095ad00-f596-438a-96c7-80757d6c8284-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80a9dfa1-mn2sk\" (UID: \"6095ad00-f596-438a-96c7-80757d6c8284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a9dfa1-mn2sk"
Oct 30 19:30:37.021813 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:37.018818    5043 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Oct 30 19:30:37.021813 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:37.018865    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq" podUID="212ce985-524a-4391-8926-63aafdc58132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Oct 30 19:30:37.022268 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:37.022245    5043 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 30 19:30:37.022441 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:37.022419    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm" podUID="03c6df78-8285-46c7-b5bd-896a0350c1de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 30 19:30:37.052188 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:37.052138    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:37.113802 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:37.113680    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:37.460819 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:37.441224    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sk69\" (UniqueName: \"kubernetes.io/projected/6095ad00-f596-438a-96c7-80757d6c8284-kube-api-access-8sk69\") pod \"maas-api-revoke-sub-80a9dfa1-mn2sk\" (UID: \"6095ad00-f596-438a-96c7-80757d6c8284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a9dfa1-mn2sk"
Oct 30 19:30:37.460819 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:37.442805    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a9dfa1-mn2sk"
Oct 30 19:30:37.594788 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:37.580866    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm"
Oct 30 19:30:37.620060 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:37.620023    5043 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 30 19:30:37.620331 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:37.620298    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm" podUID="03c6df78-8285-46c7-b5bd-896a0350c1de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 30 19:30:37.651771 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:37.626489    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:30:37.684968 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:37.684909    5043 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 19:30:37.685269 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:37.685239    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv" podUID="63688e51-3c1c-41ed-af40-4e2aa0e98c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 19:30:37.691763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:37.688467    5043 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Oct 30 19:30:37.767840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:37.745045    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq" podUID="212ce985-524a-4391-8926-63aafdc58132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Oct 30 19:30:37.767840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:37.703685    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": context deadline exceeded"
Oct 30 19:30:37.767840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:37.701869    5043 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 30 19:30:37.767840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:37.745161    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh" podUID="0aa81e41-151e-47b9-8a09-f4cf20a52dad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 30 19:30:37.814426 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:37.808044    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:37.906778 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:37.841156    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:37.906778 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:37.841353    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:30:37.906778 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:37.900654    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": context deadline exceeded"
Oct 30 19:30:37.933988 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:37.933236    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:38.093073 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:38.081546    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:38.093073 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:38.091793    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 30 19:30:38.093073 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:38.092125    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:30:38.113306 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:38.093918    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:30:38.113306 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:38.103095    5043 generic.go:356] "Generic (PLEG): container finished" podID="f42c708e-e9b6-4cd9-9cd4-090a55157421" containerID="30de05097ec1b9339ebde00cd70dbf95e988c97f28557320bbdf2482d1e87ac5" exitCode=0
Oct 30 19:30:38.113306 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:38.105096    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff807b2-dtrbk" event={"ID":"f42c708e-e9b6-4cd9-9cd4-090a55157421","Type":"ContainerDied","Data":"30de05097ec1b9339ebde00cd70dbf95e988c97f28557320bbdf2482d1e87ac5"}
Oct 30 19:30:38.302242 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:38.119867    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:38.302242 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:38.120429    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:30:38.715000 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:38.599704    5043 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 19:30:38.715278 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:38.715242    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv" podUID="63688e51-3c1c-41ed-af40-4e2aa0e98c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 19:30:38.715424 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:38.652550    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:38.715866 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:38.619887    5043 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 30 19:30:38.762874 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:38.762821    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm" podUID="03c6df78-8285-46c7-b5bd-896a0350c1de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 30 19:30:38.793762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:38.669666    5043 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Oct 30 19:30:38.794113 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:38.794073    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq" podUID="212ce985-524a-4391-8926-63aafdc58132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Oct 30 19:30:38.794256 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:38.692411    5043 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 30 19:30:39.026541 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:38.831220    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh" podUID="0aa81e41-151e-47b9-8a09-f4cf20a52dad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 30 19:30:39.026541 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:38.880835    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:39.026541 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:38.881120    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:30:39.075664 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:39.075620    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded"
Oct 30 19:30:39.081931 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:39.081895    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:39.094139 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:39.085721    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:30:39.094139 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:39.085808    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:30:39.094139 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:39.085847    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:30:39.094139 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:39.085883    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"
Oct 30 19:30:39.094139 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:39.085917    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:30:39.094957 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:39.094926    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:39.108451 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:39.108414    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:30:39.112845 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:39.112803    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:39.116963 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:39.116932    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded"
Oct 30 19:30:39.120855 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:39.120826    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 30 19:30:39.121769 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:39.121728    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:30:39.129470 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:39.122273    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:30:39.129703 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:39.129685    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:30:39.129905 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:39.129893    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"
Oct 30 19:30:39.173632 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:39.130352    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:39.371665 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:39.230684    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff807b2-dtrbk" podStartSLOduration=12.230662874 podStartE2EDuration="12.230662874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:37.029668578 +0000 UTC m=+856.540728561" watchObservedRunningTime="2026-10-30 19:30:39.230662874 +0000 UTC m=+858.741722843"
Oct 30 19:30:39.521769 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:39.515447    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:30:39.623989 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:39.623903    5043 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 19:30:39.640821 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:39.624153    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:30:39.722615 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:39.722526    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv" podUID="63688e51-3c1c-41ed-af40-4e2aa0e98c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 19:30:39.862837 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:39.798075    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:30:39.862837 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:39.821441    5043 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Oct 30 19:30:39.862837 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:39.821508    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq" podUID="212ce985-524a-4391-8926-63aafdc58132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Oct 30 19:30:39.862837 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:39.858921    5043 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 30 19:30:39.862837 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:39.858977    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm" podUID="03c6df78-8285-46c7-b5bd-896a0350c1de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 30 19:30:39.940820 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:39.893095    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:30:39.940820 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:39.910853    5043 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 30 19:30:39.940820 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:39.910912    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh" podUID="0aa81e41-151e-47b9-8a09-f4cf20a52dad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 30 19:30:39.971649 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:39.971618    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-235d91c2-bnzkl"
Oct 30 19:30:40.101919 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:40.101873    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:40.205666 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:40.120593    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:40.205666 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:40.120949    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:40.205666 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:40.124874    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:30:40.205666 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:40.125881    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:40.205666 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:40.160221    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-235d91c2-bnzkl" event={"ID":"277eda52-9140-4bc2-aacd-4b383b4ca7c5","Type":"ContainerDied","Data":"d1af545dc86f34c15d100245cfb5a49ed900134b8260e4994b88652a7b02981a"}
Oct 30 19:30:40.205666 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:40.160275    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1af545dc86f34c15d100245cfb5a49ed900134b8260e4994b88652a7b02981a"
Oct 30 19:30:40.216254 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:40.216212    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:40.246759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:40.246053    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:40.282638 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:40.282607    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/277eda52-9140-4bc2-aacd-4b383b4ca7c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/277eda52-9140-4bc2-aacd-4b383b4ca7c5-maas-api-openshift-ca\") pod \"277eda52-9140-4bc2-aacd-4b383b4ca7c5\" (UID: \"277eda52-9140-4bc2-aacd-4b383b4ca7c5\") "
Oct 30 19:30:40.283185 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:40.283162    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/277eda52-9140-4bc2-aacd-4b383b4ca7c5-kube-api-access-x9khd\" (UniqueName: \"kubernetes.io/projected/277eda52-9140-4bc2-aacd-4b383b4ca7c5-kube-api-access-x9khd\") pod \"277eda52-9140-4bc2-aacd-4b383b4ca7c5\" (UID: \"277eda52-9140-4bc2-aacd-4b383b4ca7c5\") "
Oct 30 19:30:40.283457 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:40.283439    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/277eda52-9140-4bc2-aacd-4b383b4ca7c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/277eda52-9140-4bc2-aacd-4b383b4ca7c5-maas-api-xks-ca\") pod \"277eda52-9140-4bc2-aacd-4b383b4ca7c5\" (UID: \"277eda52-9140-4bc2-aacd-4b383b4ca7c5\") "
Oct 30 19:30:40.421875 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:40.421824    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:40.440776 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:40.437962    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/277eda52-9140-4bc2-aacd-4b383b4ca7c5-maas-api-openshift-ca" pod "277eda52-9140-4bc2-aacd-4b383b4ca7c5" (UID: "277eda52-9140-4bc2-aacd-4b383b4ca7c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:40.546010 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:40.545960    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:40.615770 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:40.606189    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:30:40.710972 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:40.695313    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:40.733794 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:40.723405    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/277eda52-9140-4bc2-aacd-4b383b4ca7c5-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:40.768836 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:40.768063    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/277eda52-9140-4bc2-aacd-4b383b4ca7c5-kube-api-access-x9khd" pod "277eda52-9140-4bc2-aacd-4b383b4ca7c5" (UID: "277eda52-9140-4bc2-aacd-4b383b4ca7c5"). InnerVolumeSpecName "kube-api-access-x9khd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:40.768836 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:40.768160    5043 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Oct 30 19:30:40.768836 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:40.768203    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq" podUID="212ce985-524a-4391-8926-63aafdc58132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Oct 30 19:30:40.768836 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:40.768282    5043 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 19:30:40.768836 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:40.768303    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv" podUID="63688e51-3c1c-41ed-af40-4e2aa0e98c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 19:30:40.878805 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:40.818404    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:40.976957 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:40.946841    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:30:40.978374 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:40.978339    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9khd\" (UniqueName: \"kubernetes.io/projected/277eda52-9140-4bc2-aacd-4b383b4ca7c5-kube-api-access-x9khd\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:41.062685 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:41.062581    5043 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 30 19:30:41.126894 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:41.126852    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh" podUID="0aa81e41-151e-47b9-8a09-f4cf20a52dad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 30 19:30:41.127463 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:41.075575    5043 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 30 19:30:41.127591 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:41.127569    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm" podUID="03c6df78-8285-46c7-b5bd-896a0350c1de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 30 19:30:41.153761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:41.129080    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:30:41.649078 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:41.649034    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:30:41.649542 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:41.181880    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:30:41.649849 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:41.223841    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:41.649982 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:41.236408    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:30:41.650170 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:41.650153    5043 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 30 19:30:41.650286 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:41.650266    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm" podUID="03c6df78-8285-46c7-b5bd-896a0350c1de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 30 19:30:41.650401 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:41.650371    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:41.659756 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:41.181846    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded"
Oct 30 19:30:41.691845 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:41.691815    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:30:41.740778 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:41.712583    5043 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 19:30:41.741053 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:41.741019    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv" podUID="63688e51-3c1c-41ed-af40-4e2aa0e98c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 19:30:41.741267 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:41.712822    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:41.741668 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:41.712776    5043 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Oct 30 19:30:41.741833 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:41.741814    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq" podUID="212ce985-524a-4391-8926-63aafdc58132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Oct 30 19:30:41.741970 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:41.741958    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:30:41.785779 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:41.751181    5043 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 30 19:30:42.051046 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:42.050956    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh" podUID="0aa81e41-151e-47b9-8a09-f4cf20a52dad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 30 19:30:42.247859 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:41.951205    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:42.513245 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:42.048867    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zt5b2" podUID="a348d1da-3b22-4650-84a5-4e2de27d37e3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:42.513245 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:41.826048    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:42.513245 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:41.833868    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/277eda52-9140-4bc2-aacd-4b383b4ca7c5-maas-api-xks-ca" pod "277eda52-9140-4bc2-aacd-4b383b4ca7c5" (UID: "277eda52-9140-4bc2-aacd-4b383b4ca7c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:30:42.593084 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:42.593040    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv"
Oct 30 19:30:42.622793 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:42.602690    5043 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 19:30:42.622793 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:42.602765    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv" podUID="63688e51-3c1c-41ed-af40-4e2aa0e98c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 19:30:42.622793 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:42.603486    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/277eda52-9140-4bc2-aacd-4b383b4ca7c5-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:42.628042 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:42.628012    5043 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 30 19:30:42.628262 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:42.628238    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm" podUID="03c6df78-8285-46c7-b5bd-896a0350c1de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 30 19:30:42.644986 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:42.644950    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq"
Oct 30 19:30:42.675009 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:42.674908    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:42.686840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:42.675232    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:42.687519 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:42.675265    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:30:42.726870 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:42.726826    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:42.729765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:42.727725    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded"
Oct 30 19:30:42.750773 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:42.730185    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 30 19:30:42.752069 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:42.743230    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:30:42.752571 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:42.743305    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:42.780782 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:42.763610    5043 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 30 19:30:42.824800 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:42.822090    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh" podUID="0aa81e41-151e-47b9-8a09-f4cf20a52dad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 30 19:30:43.567770 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:43.508597    5043 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Oct 30 19:30:43.567770 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:43.562562    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq" podUID="212ce985-524a-4391-8926-63aafdc58132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Oct 30 19:30:43.567770 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:43.565462    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/health\": context deadline exceeded"
Oct 30 19:30:43.583921 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:43.583888    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-235d91c2-bnzkl"
Oct 30 19:30:43.677819 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:43.656447    5043 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 30 19:30:43.677819 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:43.656516    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm" podUID="03c6df78-8285-46c7-b5bd-896a0350c1de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 30 19:30:43.685317 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:43.685285    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff807b2-dtrbk"
Oct 30 19:30:43.752762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:43.735924    5043 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 30 19:30:43.752762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:43.735990    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh" podUID="0aa81e41-151e-47b9-8a09-f4cf20a52dad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 30 19:30:43.752762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:43.736439    5043 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 19:30:43.752762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:43.736494    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv" podUID="63688e51-3c1c-41ed-af40-4e2aa0e98c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 19:30:43.773015 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:43.767203    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:30:43.804142 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:43.646851    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded"
Oct 30 19:30:43.885451 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:43.885338    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:30:43.906762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:43.893226    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:43.942012 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:43.902897    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:30:43.942336 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:43.903448    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 30 19:30:43.942533 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:43.922281    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": context deadline exceeded"
Oct 30 19:30:43.967273 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:43.967231    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded"
Oct 30 19:30:43.967707 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:43.967681    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:30:43.975057 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:43.975033    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:30:44.155826 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:44.150877    5043 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Oct 30 19:30:44.180761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:44.156073    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq" podUID="212ce985-524a-4391-8926-63aafdc58132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Oct 30 19:30:44.355762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:44.337889    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded"
Oct 30 19:30:44.355762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:44.338421    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f42c708e-e9b6-4cd9-9cd4-090a55157421-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f42c708e-e9b6-4cd9-9cd4-090a55157421-maas-api-xks-ca\") pod \"f42c708e-e9b6-4cd9-9cd4-090a55157421\" (UID: \"f42c708e-e9b6-4cd9-9cd4-090a55157421\") "
Oct 30 19:30:44.355762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:44.338504    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f42c708e-e9b6-4cd9-9cd4-090a55157421-kube-api-access-k7qzx\" (UniqueName: \"kubernetes.io/projected/f42c708e-e9b6-4cd9-9cd4-090a55157421-kube-api-access-k7qzx\") pod \"f42c708e-e9b6-4cd9-9cd4-090a55157421\" (UID: \"f42c708e-e9b6-4cd9-9cd4-090a55157421\") "
Oct 30 19:30:44.355762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:44.338693    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f42c708e-e9b6-4cd9-9cd4-090a55157421-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f42c708e-e9b6-4cd9-9cd4-090a55157421-maas-api-openshift-ca\") pod \"f42c708e-e9b6-4cd9-9cd4-090a55157421\" (UID: \"f42c708e-e9b6-4cd9-9cd4-090a55157421\") "
Oct 30 19:30:44.473494 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:44.473438    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f42c708e-e9b6-4cd9-9cd4-090a55157421-kube-api-access-k7qzx" pod "f42c708e-e9b6-4cd9-9cd4-090a55157421" (UID: "f42c708e-e9b6-4cd9-9cd4-090a55157421"). InnerVolumeSpecName "kube-api-access-k7qzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:44.531765 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:30:44.522053    5043 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.875s"
Oct 30 19:30:44.557328 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:44.557294    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a9dfa1-mn2sk" event={"ID":"6095ad00-f596-438a-96c7-80757d6c8284","Type":"ContainerStarted","Data":"96a4b1ab794ccf637f87c9230740e3de90282b101a939b22708ec3df81d5010f"}
Oct 30 19:30:44.557808 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:44.557730    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:30:44.560074 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:44.557945    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:30:44.560237 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:44.560222    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm"
Oct 30 19:30:44.560360 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:44.560348    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh"
Oct 30 19:30:44.560490 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:44.560479    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:30:44.628938 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:44.548635    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f42c708e-e9b6-4cd9-9cd4-090a55157421-maas-api-openshift-ca" pod "f42c708e-e9b6-4cd9-9cd4-090a55157421" (UID: "f42c708e-e9b6-4cd9-9cd4-090a55157421"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:44.628938 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:44.598994    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7qzx\" (UniqueName: \"kubernetes.io/projected/f42c708e-e9b6-4cd9-9cd4-090a55157421-kube-api-access-k7qzx\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:44.628938 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:44.599020    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f42c708e-e9b6-4cd9-9cd4-090a55157421-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:44.758100 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:44.758005    5043 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Oct 30 19:30:44.766170 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:44.766127    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq" podUID="212ce985-524a-4391-8926-63aafdc58132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Oct 30 19:30:44.776595 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:44.776565    5043 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 30 19:30:44.786019 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:44.785961    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm" podUID="03c6df78-8285-46c7-b5bd-896a0350c1de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 30 19:30:44.803256 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:44.802095    5043 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 30 19:30:44.803256 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:44.802165    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh" podUID="0aa81e41-151e-47b9-8a09-f4cf20a52dad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 30 19:30:44.895606 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:44.895565    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:44.901166 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:44.900672    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:44.901166 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:44.900801    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:30:44.945776 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:44.945048    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:44.945776 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:44.945365    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:30:44.945776 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:44.945562    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:30:44.968920 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:44.968876    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:44.970404 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:44.970374    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded"
Oct 30 19:30:45.036583 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:45.034709    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:30:45.077350 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:45.071450    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f42c708e-e9b6-4cd9-9cd4-090a55157421-maas-api-xks-ca" pod "f42c708e-e9b6-4cd9-9cd4-090a55157421" (UID: "f42c708e-e9b6-4cd9-9cd4-090a55157421"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:30:45.177058 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:45.173976    5043 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Oct 30 19:30:45.177058 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:45.174031    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv" podUID="63688e51-3c1c-41ed-af40-4e2aa0e98c9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Oct 30 19:30:45.219617 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:45.215257    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f42c708e-e9b6-4cd9-9cd4-090a55157421-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:45.304663 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:45.282658    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:30:45.318394 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:45.318358    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:30:45.318565 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:45.318420    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"
Oct 30 19:30:45.318565 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:45.318460    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80a9dfa1-mn2sk"]
Oct 30 19:30:45.348407 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:45.348106    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:30:45.457192 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:45.457164    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:30:45.495761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:45.486573    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a9dfa1-mn2sk" event={"ID":"6095ad00-f596-438a-96c7-80757d6c8284","Type":"ContainerStarted","Data":"3d7cebf2c5881af085ccfbdfd8b1b13d028224eb99c9b18abe4b12b498c83d4d"}
Oct 30 19:30:45.627630 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:45.627547    5043 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 30 19:30:45.627887 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:45.627860    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm" podUID="03c6df78-8285-46c7-b5bd-896a0350c1de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 30 19:30:45.660617 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:45.660151    5043 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Oct 30 19:30:45.660617 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:45.660211    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq" podUID="212ce985-524a-4391-8926-63aafdc58132" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Oct 30 19:30:45.660617 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:45.660529    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff807b2-dtrbk" event={"ID":"f42c708e-e9b6-4cd9-9cd4-090a55157421","Type":"ContainerDied","Data":"53531fec56ee6b2afc6b17c756c6e2f122d082ab22addd149b6601827dcbb267"}
Oct 30 19:30:45.660617 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:45.660579    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53531fec56ee6b2afc6b17c756c6e2f122d082ab22addd149b6601827dcbb267"
Oct 30 19:30:45.662983 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:45.661405    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff807b2-dtrbk"
Oct 30 19:30:45.687810 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:45.685319    5043 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 30 19:30:45.687810 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:45.685373    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh" podUID="0aa81e41-151e-47b9-8a09-f4cf20a52dad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 30 19:30:45.849753 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:45.847901    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:30:45.962817 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:45.962785    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:30:45.967128 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:45.967101    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"
Oct 30 19:30:46.395183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:46.377071    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv"
Oct 30 19:30:46.400427 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:46.400303    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv"
Oct 30 19:30:46.402329 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:30:46.402296    5043 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod277eda52_9140_4bc2_aacd_4b383b4ca7c5.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6095ad00_f596_438a_96c7_80757d6c8284.slice/crio-conmon-3d7cebf2c5881af085ccfbdfd8b1b13d028224eb99c9b18abe4b12b498c83d4d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6095ad00_f596_438a_96c7_80757d6c8284.slice/crio-3d7cebf2c5881af085ccfbdfd8b1b13d028224eb99c9b18abe4b12b498c83d4d.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 19:30:46.410356 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:46.410188    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:30:46.495392 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:46.495348    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5166f1e7-j5jqp"]
Oct 30 19:30:46.496920 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:46.496879    5043 cpu_manager.go:414] "containerMap: removing container" podUID="f42c708e-e9b6-4cd9-9cd4-090a55157421" containerName="revoke-keys"
Oct 30 19:30:46.497090 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:46.497074    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f42c708e-e9b6-4cd9-9cd4-090a55157421" containerName="revoke-keys"
Oct 30 19:30:46.497232 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:46.497219    5043 cpu_manager.go:414] "containerMap: removing container" podUID="277eda52-9140-4bc2-aacd-4b383b4ca7c5" containerName="revoke-keys"
Oct 30 19:30:46.497330 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:46.497316    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="277eda52-9140-4bc2-aacd-4b383b4ca7c5" containerName="revoke-keys"
Oct 30 19:30:46.497587 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:46.497570    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="277eda52-9140-4bc2-aacd-4b383b4ca7c5" containerName="revoke-keys"
Oct 30 19:30:46.497722 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:46.497710    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="f42c708e-e9b6-4cd9-9cd4-090a55157421" containerName="revoke-keys"
Oct 30 19:30:46.504642 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:46.504624    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5166f1e7-j5jqp"
Oct 30 19:30:46.541024 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:46.540972    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5166f1e7-j5jqp"]
Oct 30 19:30:46.603881 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:46.603839    5043 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 30 19:30:46.604298 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:46.604247    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm" podUID="03c6df78-8285-46c7-b5bd-896a0350c1de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 30 19:30:46.661481 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:46.661387    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swfmq\" (UniqueName: \"kubernetes.io/projected/d4c85361-fb0b-49e4-8f86-3fb92a60ab61-kube-api-access-swfmq\") pod \"maas-api-revoke-sub-5166f1e7-j5jqp\" (UID: \"d4c85361-fb0b-49e4-8f86-3fb92a60ab61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5166f1e7-j5jqp"
Oct 30 19:30:46.662169 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:46.662145    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4c85361-fb0b-49e4-8f86-3fb92a60ab61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5166f1e7-j5jqp\" (UID: \"d4c85361-fb0b-49e4-8f86-3fb92a60ab61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5166f1e7-j5jqp"
Oct 30 19:30:46.662401 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:46.662382    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4c85361-fb0b-49e4-8f86-3fb92a60ab61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5166f1e7-j5jqp\" (UID: \"d4c85361-fb0b-49e4-8f86-3fb92a60ab61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5166f1e7-j5jqp"
Oct 30 19:30:46.687130 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:46.687024    5043 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 30 19:30:46.687384 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:46.687351    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh" podUID="0aa81e41-151e-47b9-8a09-f4cf20a52dad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 30 19:30:46.696182 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:46.694295    5043 generic.go:356] "Generic (PLEG): container finished" podID="6095ad00-f596-438a-96c7-80757d6c8284" containerID="3d7cebf2c5881af085ccfbdfd8b1b13d028224eb99c9b18abe4b12b498c83d4d" exitCode=0
Oct 30 19:30:46.696182 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:46.694402    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a9dfa1-mn2sk" event={"ID":"6095ad00-f596-438a-96c7-80757d6c8284","Type":"ContainerDied","Data":"3d7cebf2c5881af085ccfbdfd8b1b13d028224eb99c9b18abe4b12b498c83d4d"}
Oct 30 19:30:46.788332 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:46.774849    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swfmq\" (UniqueName: \"kubernetes.io/projected/d4c85361-fb0b-49e4-8f86-3fb92a60ab61-kube-api-access-swfmq\") pod \"maas-api-revoke-sub-5166f1e7-j5jqp\" (UID: \"d4c85361-fb0b-49e4-8f86-3fb92a60ab61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5166f1e7-j5jqp"
Oct 30 19:30:46.788332 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:46.774991    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4c85361-fb0b-49e4-8f86-3fb92a60ab61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5166f1e7-j5jqp\" (UID: \"d4c85361-fb0b-49e4-8f86-3fb92a60ab61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5166f1e7-j5jqp"
Oct 30 19:30:46.788332 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:46.775128    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4c85361-fb0b-49e4-8f86-3fb92a60ab61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5166f1e7-j5jqp\" (UID: \"d4c85361-fb0b-49e4-8f86-3fb92a60ab61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5166f1e7-j5jqp"
Oct 30 19:30:46.864756 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:46.863156    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4c85361-fb0b-49e4-8f86-3fb92a60ab61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5166f1e7-j5jqp\" (UID: \"d4c85361-fb0b-49e4-8f86-3fb92a60ab61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5166f1e7-j5jqp"
Oct 30 19:30:47.038290 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:47.038241    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4c85361-fb0b-49e4-8f86-3fb92a60ab61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5166f1e7-j5jqp\" (UID: \"d4c85361-fb0b-49e4-8f86-3fb92a60ab61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5166f1e7-j5jqp"
Oct 30 19:30:47.105792 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:47.105761    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2ae6628f-54ff895c9d-4d9pk"]
Oct 30 19:30:47.117363 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:47.117337    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2ae6628f-54ff895c9d-4d9pk"
Oct 30 19:30:47.141610 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:47.141580    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-2ae6628f\""
Oct 30 19:30:47.142071 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:47.142050    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-2ae6628f-dockercfg-j5d6s\""
Oct 30 19:30:47.174301 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:47.174198    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swfmq\" (UniqueName: \"kubernetes.io/projected/d4c85361-fb0b-49e4-8f86-3fb92a60ab61-kube-api-access-swfmq\") pod \"maas-api-revoke-sub-5166f1e7-j5jqp\" (UID: \"d4c85361-fb0b-49e4-8f86-3fb92a60ab61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5166f1e7-j5jqp"
Oct 30 19:30:47.198888 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:47.198864    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2ae6628f-54ff895c9d-4d9pk"]
Oct 30 19:30:47.212528 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:47.212493    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff4784f6-f68b-4183-bd3c-d52592a374c4-plugins-config-volume\") pod \"payload-processing-e2e-ait-2ae6628f-54ff895c9d-4d9pk\" (UID: \"ff4784f6-f68b-4183-bd3c-d52592a374c4\") " pod="openshift-ingress/payload-processing-e2e-ait-2ae6628f-54ff895c9d-4d9pk"
Oct 30 19:30:47.212771 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:47.212751    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvmc9\" (UniqueName: \"kubernetes.io/projected/ff4784f6-f68b-4183-bd3c-d52592a374c4-kube-api-access-dvmc9\") pod \"payload-processing-e2e-ait-2ae6628f-54ff895c9d-4d9pk\" (UID: \"ff4784f6-f68b-4183-bd3c-d52592a374c4\") " pod="openshift-ingress/payload-processing-e2e-ait-2ae6628f-54ff895c9d-4d9pk"
Oct 30 19:30:47.222818 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:47.222782    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b25eed1-4dkn6"]
Oct 30 19:30:47.228640 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:47.228622    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b25eed1-4dkn6"
Oct 30 19:30:47.240841 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:47.240809    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b25eed1-4dkn6"]
Oct 30 19:30:47.320913 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:47.314723    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cgxc\" (UniqueName: \"kubernetes.io/projected/1a377cda-465d-45ca-bdd3-5bb94c19ed24-kube-api-access-2cgxc\") pod \"maas-api-revoke-sub-2b25eed1-4dkn6\" (UID: \"1a377cda-465d-45ca-bdd3-5bb94c19ed24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b25eed1-4dkn6"
Oct 30 19:30:47.320913 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:47.314868    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff4784f6-f68b-4183-bd3c-d52592a374c4-plugins-config-volume\") pod \"payload-processing-e2e-ait-2ae6628f-54ff895c9d-4d9pk\" (UID: \"ff4784f6-f68b-4183-bd3c-d52592a374c4\") " pod="openshift-ingress/payload-processing-e2e-ait-2ae6628f-54ff895c9d-4d9pk"
Oct 30 19:30:47.320913 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:47.314955    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a377cda-465d-45ca-bdd3-5bb94c19ed24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b25eed1-4dkn6\" (UID: \"1a377cda-465d-45ca-bdd3-5bb94c19ed24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b25eed1-4dkn6"
Oct 30 19:30:47.320913 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:47.315771    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff4784f6-f68b-4183-bd3c-d52592a374c4-plugins-config-volume\") pod \"payload-processing-e2e-ait-2ae6628f-54ff895c9d-4d9pk\" (UID: \"ff4784f6-f68b-4183-bd3c-d52592a374c4\") " pod="openshift-ingress/payload-processing-e2e-ait-2ae6628f-54ff895c9d-4d9pk"
Oct 30 19:30:47.320913 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:47.315835    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvmc9\" (UniqueName: \"kubernetes.io/projected/ff4784f6-f68b-4183-bd3c-d52592a374c4-kube-api-access-dvmc9\") pod \"payload-processing-e2e-ait-2ae6628f-54ff895c9d-4d9pk\" (UID: \"ff4784f6-f68b-4183-bd3c-d52592a374c4\") " pod="openshift-ingress/payload-processing-e2e-ait-2ae6628f-54ff895c9d-4d9pk"
Oct 30 19:30:47.320913 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:47.317033    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a377cda-465d-45ca-bdd3-5bb94c19ed24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b25eed1-4dkn6\" (UID: \"1a377cda-465d-45ca-bdd3-5bb94c19ed24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b25eed1-4dkn6"
Oct 30 19:30:47.434651 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:47.429609    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvmc9\" (UniqueName: \"kubernetes.io/projected/ff4784f6-f68b-4183-bd3c-d52592a374c4-kube-api-access-dvmc9\") pod \"payload-processing-e2e-ait-2ae6628f-54ff895c9d-4d9pk\" (UID: \"ff4784f6-f68b-4183-bd3c-d52592a374c4\") " pod="openshift-ingress/payload-processing-e2e-ait-2ae6628f-54ff895c9d-4d9pk"
Oct 30 19:30:47.435579 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:47.435546    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cgxc\" (UniqueName: \"kubernetes.io/projected/1a377cda-465d-45ca-bdd3-5bb94c19ed24-kube-api-access-2cgxc\") pod \"maas-api-revoke-sub-2b25eed1-4dkn6\" (UID: \"1a377cda-465d-45ca-bdd3-5bb94c19ed24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b25eed1-4dkn6"
Oct 30 19:30:47.436344 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:47.436318    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a377cda-465d-45ca-bdd3-5bb94c19ed24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b25eed1-4dkn6\" (UID: \"1a377cda-465d-45ca-bdd3-5bb94c19ed24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b25eed1-4dkn6"
Oct 30 19:30:47.437505 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:47.437482    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a377cda-465d-45ca-bdd3-5bb94c19ed24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b25eed1-4dkn6\" (UID: \"1a377cda-465d-45ca-bdd3-5bb94c19ed24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b25eed1-4dkn6"
Oct 30 19:30:47.438119 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:47.437334    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a377cda-465d-45ca-bdd3-5bb94c19ed24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b25eed1-4dkn6\" (UID: \"1a377cda-465d-45ca-bdd3-5bb94c19ed24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b25eed1-4dkn6"
Oct 30 19:30:47.441687 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:47.441470    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5166f1e7-j5jqp"
Oct 30 19:30:47.442240 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:47.442190    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a377cda-465d-45ca-bdd3-5bb94c19ed24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b25eed1-4dkn6\" (UID: \"1a377cda-465d-45ca-bdd3-5bb94c19ed24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b25eed1-4dkn6"
Oct 30 19:30:47.449973 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:47.448145    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq"
Oct 30 19:30:47.454251 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:47.454154    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq"
Oct 30 19:30:47.487762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:47.455679    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cgxc\" (UniqueName: \"kubernetes.io/projected/1a377cda-465d-45ca-bdd3-5bb94c19ed24-kube-api-access-2cgxc\") pod \"maas-api-revoke-sub-2b25eed1-4dkn6\" (UID: \"1a377cda-465d-45ca-bdd3-5bb94c19ed24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b25eed1-4dkn6"
Oct 30 19:30:47.526132 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:47.512625    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2ae6628f-54ff895c9d-4d9pk"
Oct 30 19:30:47.539582 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:47.539553    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a9dfa1-mn2sk"
Oct 30 19:30:47.567986 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:47.567906    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b25eed1-4dkn6"
Oct 30 19:30:47.647885 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:47.645554    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6095ad00-f596-438a-96c7-80757d6c8284-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6095ad00-f596-438a-96c7-80757d6c8284-maas-api-openshift-ca\") pod \"6095ad00-f596-438a-96c7-80757d6c8284\" (UID: \"6095ad00-f596-438a-96c7-80757d6c8284\") "
Oct 30 19:30:47.647885 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:47.645612    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6095ad00-f596-438a-96c7-80757d6c8284-kube-api-access-8sk69\" (UniqueName: \"kubernetes.io/projected/6095ad00-f596-438a-96c7-80757d6c8284-kube-api-access-8sk69\") pod \"6095ad00-f596-438a-96c7-80757d6c8284\" (UID: \"6095ad00-f596-438a-96c7-80757d6c8284\") "
Oct 30 19:30:47.650528 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:47.650476    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6095ad00-f596-438a-96c7-80757d6c8284-maas-api-openshift-ca" pod "6095ad00-f596-438a-96c7-80757d6c8284" (UID: "6095ad00-f596-438a-96c7-80757d6c8284"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:47.652847 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:47.651991    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6095ad00-f596-438a-96c7-80757d6c8284-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6095ad00-f596-438a-96c7-80757d6c8284-maas-api-xks-ca\") pod \"6095ad00-f596-438a-96c7-80757d6c8284\" (UID: \"6095ad00-f596-438a-96c7-80757d6c8284\") "
Oct 30 19:30:47.653448 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:47.653121    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6095ad00-f596-438a-96c7-80757d6c8284-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:47.667485 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:47.667350    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6095ad00-f596-438a-96c7-80757d6c8284-maas-api-xks-ca" pod "6095ad00-f596-438a-96c7-80757d6c8284" (UID: "6095ad00-f596-438a-96c7-80757d6c8284"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:30:47.670803 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:47.668333    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6095ad00-f596-438a-96c7-80757d6c8284-kube-api-access-8sk69" pod "6095ad00-f596-438a-96c7-80757d6c8284" (UID: "6095ad00-f596-438a-96c7-80757d6c8284"). InnerVolumeSpecName "kube-api-access-8sk69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:47.746999 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:47.744423    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a9dfa1-mn2sk"
Oct 30 19:30:47.746999 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:47.746913    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a9dfa1-mn2sk" event={"ID":"6095ad00-f596-438a-96c7-80757d6c8284","Type":"ContainerDied","Data":"96a4b1ab794ccf637f87c9230740e3de90282b101a939b22708ec3df81d5010f"}
Oct 30 19:30:47.746999 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:47.746967    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96a4b1ab794ccf637f87c9230740e3de90282b101a939b22708ec3df81d5010f"
Oct 30 19:30:47.755164 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:47.755029    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sk69\" (UniqueName: \"kubernetes.io/projected/6095ad00-f596-438a-96c7-80757d6c8284-kube-api-access-8sk69\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:47.755164 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:47.755103    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6095ad00-f596-438a-96c7-80757d6c8284-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:48.193630 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:48.164010    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm"
Oct 30 19:30:48.201626 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:48.201558    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5166f1e7-j5jqp"]
Oct 30 19:30:48.219986 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:48.219844    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm"
Oct 30 19:30:48.329084 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:30:48.329042    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff4784f6_f68b_4183_bd3c_d52592a374c4.slice/crio-20251be92ee14ff5dbbc149d59c44f860059cd5e858c212e3e1cc7fc683466e3 WatchSource:0}: Error finding container 20251be92ee14ff5dbbc149d59c44f860059cd5e858c212e3e1cc7fc683466e3: Status 404 returned error can't find the container with id 20251be92ee14ff5dbbc149d59c44f860059cd5e858c212e3e1cc7fc683466e3
Oct 30 19:30:48.333656 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:48.333632    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh"
Oct 30 19:30:48.333852 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:48.333836    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2ae6628f-54ff895c9d-4d9pk"]
Oct 30 19:30:48.336991 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:48.336969    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh"
Oct 30 19:30:48.446100 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:48.446065    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b25eed1-4dkn6"]
Oct 30 19:30:48.464303 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:30:48.464277    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a377cda_465d_45ca_bdd3_5bb94c19ed24.slice/crio-1472b8f76a464e52b4f7810aeb9fca392c8b1975090ee847e2f1cdf324933d9c WatchSource:0}: Error finding container 1472b8f76a464e52b4f7810aeb9fca392c8b1975090ee847e2f1cdf324933d9c: Status 404 returned error can't find the container with id 1472b8f76a464e52b4f7810aeb9fca392c8b1975090ee847e2f1cdf324933d9c
Oct 30 19:30:48.682230 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:48.682199    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c91cb764-knbd8"]
Oct 30 19:30:48.682953 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:48.682909    5043 cpu_manager.go:414] "containerMap: removing container" podUID="6095ad00-f596-438a-96c7-80757d6c8284" containerName="revoke-keys"
Oct 30 19:30:48.682953 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:48.682925    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6095ad00-f596-438a-96c7-80757d6c8284" containerName="revoke-keys"
Oct 30 19:30:48.683057 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:48.683048    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="6095ad00-f596-438a-96c7-80757d6c8284" containerName="revoke-keys"
Oct 30 19:30:48.688261 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:48.688226    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91cb764-knbd8"
Oct 30 19:30:48.693398 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:48.693373    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c91cb764-knbd8"]
Oct 30 19:30:48.702949 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:48.701677    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1c894c4-4433-4736-b57e-b6bf5db5e698-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c91cb764-knbd8\" (UID: \"a1c894c4-4433-4736-b57e-b6bf5db5e698\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91cb764-knbd8"
Oct 30 19:30:48.702949 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:48.701783    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnm8f\" (UniqueName: \"kubernetes.io/projected/a1c894c4-4433-4736-b57e-b6bf5db5e698-kube-api-access-fnm8f\") pod \"maas-api-revoke-sub-c91cb764-knbd8\" (UID: \"a1c894c4-4433-4736-b57e-b6bf5db5e698\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91cb764-knbd8"
Oct 30 19:30:48.702949 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:48.701828    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1c894c4-4433-4736-b57e-b6bf5db5e698-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c91cb764-knbd8\" (UID: \"a1c894c4-4433-4736-b57e-b6bf5db5e698\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91cb764-knbd8"
Oct 30 19:30:48.755645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:48.755607    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2ae6628f-54ff895c9d-4d9pk" event={"ID":"ff4784f6-f68b-4183-bd3c-d52592a374c4","Type":"ContainerStarted","Data":"26be335961aebc998e1b13698225cbe47008bbd1a619a856ff25d040ba3abf10"}
Oct 30 19:30:48.755861 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:48.755667    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2ae6628f-54ff895c9d-4d9pk" event={"ID":"ff4784f6-f68b-4183-bd3c-d52592a374c4","Type":"ContainerStarted","Data":"20251be92ee14ff5dbbc149d59c44f860059cd5e858c212e3e1cc7fc683466e3"}
Oct 30 19:30:48.755861 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:48.755697    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-2ae6628f-54ff895c9d-4d9pk"
Oct 30 19:30:48.757470 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:48.757436    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b25eed1-4dkn6" event={"ID":"1a377cda-465d-45ca-bdd3-5bb94c19ed24","Type":"ContainerStarted","Data":"1472b8f76a464e52b4f7810aeb9fca392c8b1975090ee847e2f1cdf324933d9c"}
Oct 30 19:30:48.759200 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:48.759169    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5166f1e7-j5jqp" event={"ID":"d4c85361-fb0b-49e4-8f86-3fb92a60ab61","Type":"ContainerStarted","Data":"dfce87515da3b0b1b72100aeb662ba2ae7ac7d9cacbd7bc443bb28943cdfe4a0"}
Oct 30 19:30:48.775935 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:48.775895    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-2ae6628f-54ff895c9d-4d9pk" podStartSLOduration=2.775882153 podStartE2EDuration="2.775882153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:48.771191586 +0000 UTC m=+868.282251552" watchObservedRunningTime="2026-10-30 19:30:48.775882153 +0000 UTC m=+868.286942127"
Oct 30 19:30:48.803585 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:48.803553    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnm8f\" (UniqueName: \"kubernetes.io/projected/a1c894c4-4433-4736-b57e-b6bf5db5e698-kube-api-access-fnm8f\") pod \"maas-api-revoke-sub-c91cb764-knbd8\" (UID: \"a1c894c4-4433-4736-b57e-b6bf5db5e698\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91cb764-knbd8"
Oct 30 19:30:48.803719 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:48.803637    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1c894c4-4433-4736-b57e-b6bf5db5e698-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c91cb764-knbd8\" (UID: \"a1c894c4-4433-4736-b57e-b6bf5db5e698\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91cb764-knbd8"
Oct 30 19:30:48.804085 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:48.804060    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1c894c4-4433-4736-b57e-b6bf5db5e698-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c91cb764-knbd8\" (UID: \"a1c894c4-4433-4736-b57e-b6bf5db5e698\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91cb764-knbd8"
Oct 30 19:30:48.804856 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:48.804833    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1c894c4-4433-4736-b57e-b6bf5db5e698-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c91cb764-knbd8\" (UID: \"a1c894c4-4433-4736-b57e-b6bf5db5e698\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91cb764-knbd8"
Oct 30 19:30:48.806911 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:48.806890    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1c894c4-4433-4736-b57e-b6bf5db5e698-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c91cb764-knbd8\" (UID: \"a1c894c4-4433-4736-b57e-b6bf5db5e698\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91cb764-knbd8"
Oct 30 19:30:48.811380 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:48.811354    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnm8f\" (UniqueName: \"kubernetes.io/projected/a1c894c4-4433-4736-b57e-b6bf5db5e698-kube-api-access-fnm8f\") pod \"maas-api-revoke-sub-c91cb764-knbd8\" (UID: \"a1c894c4-4433-4736-b57e-b6bf5db5e698\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91cb764-knbd8"
Oct 30 19:30:49.008171 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:49.008134    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91cb764-knbd8"
Oct 30 19:30:49.308825 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:49.308797    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c91cb764-knbd8"]
Oct 30 19:30:49.334634 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:30:49.334601    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1c894c4_4433_4736_b57e_b6bf5db5e698.slice/crio-119f3c4b7612cc10ec8d0816e0afdffbfa17209abd569a95d4589c5b0832ccaa WatchSource:0}: Error finding container 119f3c4b7612cc10ec8d0816e0afdffbfa17209abd569a95d4589c5b0832ccaa: Status 404 returned error can't find the container with id 119f3c4b7612cc10ec8d0816e0afdffbfa17209abd569a95d4589c5b0832ccaa
Oct 30 19:30:49.774881 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:49.774854    5043 generic.go:356] "Generic (PLEG): container finished" podID="1a377cda-465d-45ca-bdd3-5bb94c19ed24" containerID="1705f002e024c36664dde4d28c05ace7afcb0309f2b82169f0a412b0dab2869b" exitCode=0
Oct 30 19:30:49.775428 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:49.774947    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b25eed1-4dkn6" event={"ID":"1a377cda-465d-45ca-bdd3-5bb94c19ed24","Type":"ContainerDied","Data":"1705f002e024c36664dde4d28c05ace7afcb0309f2b82169f0a412b0dab2869b"}
Oct 30 19:30:49.777018 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:49.776992    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91cb764-knbd8" event={"ID":"a1c894c4-4433-4736-b57e-b6bf5db5e698","Type":"ContainerStarted","Data":"119f3c4b7612cc10ec8d0816e0afdffbfa17209abd569a95d4589c5b0832ccaa"}
Oct 30 19:30:49.779083 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:49.779054    5043 generic.go:356] "Generic (PLEG): container finished" podID="d4c85361-fb0b-49e4-8f86-3fb92a60ab61" containerID="0ecf41f512dd96fa1fba76b270d68443bc3faed4e59292f2daa9301bae6bd671" exitCode=0
Oct 30 19:30:49.779198 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:49.779155    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5166f1e7-j5jqp" event={"ID":"d4c85361-fb0b-49e4-8f86-3fb92a60ab61","Type":"ContainerDied","Data":"0ecf41f512dd96fa1fba76b270d68443bc3faed4e59292f2daa9301bae6bd671"}
Oct 30 19:30:49.813876 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:49.813822    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91cb764-knbd8" podStartSLOduration=1.8138059069999999 podStartE2EDuration="1.813805907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:49.812930282 +0000 UTC m=+869.323990254" watchObservedRunningTime="2026-10-30 19:30:49.813805907 +0000 UTC m=+869.324865879"
Oct 30 19:30:50.792128 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:50.792088    5043 generic.go:356] "Generic (PLEG): container finished" podID="a1c894c4-4433-4736-b57e-b6bf5db5e698" containerID="d4c089655e949581d4ef453c301ac4802f81fea906ed0fdc8b15d56d4def256f" exitCode=0
Oct 30 19:30:50.792526 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:50.792162    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91cb764-knbd8" event={"ID":"a1c894c4-4433-4736-b57e-b6bf5db5e698","Type":"ContainerDied","Data":"d4c089655e949581d4ef453c301ac4802f81fea906ed0fdc8b15d56d4def256f"}
Oct 30 19:30:51.306046 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:51.306024    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b25eed1-4dkn6"
Oct 30 19:30:51.342371 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:51.342328    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5166f1e7-j5jqp"
Oct 30 19:30:51.466207 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:51.466121    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a377cda-465d-45ca-bdd3-5bb94c19ed24-kube-api-access-2cgxc\" (UniqueName: \"kubernetes.io/projected/1a377cda-465d-45ca-bdd3-5bb94c19ed24-kube-api-access-2cgxc\") pod \"1a377cda-465d-45ca-bdd3-5bb94c19ed24\" (UID: \"1a377cda-465d-45ca-bdd3-5bb94c19ed24\") "
Oct 30 19:30:51.466399 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:51.466243    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4c85361-fb0b-49e4-8f86-3fb92a60ab61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4c85361-fb0b-49e4-8f86-3fb92a60ab61-maas-api-openshift-ca\") pod \"d4c85361-fb0b-49e4-8f86-3fb92a60ab61\" (UID: \"d4c85361-fb0b-49e4-8f86-3fb92a60ab61\") "
Oct 30 19:30:51.466399 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:51.466305    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a377cda-465d-45ca-bdd3-5bb94c19ed24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a377cda-465d-45ca-bdd3-5bb94c19ed24-maas-api-openshift-ca\") pod \"1a377cda-465d-45ca-bdd3-5bb94c19ed24\" (UID: \"1a377cda-465d-45ca-bdd3-5bb94c19ed24\") "
Oct 30 19:30:51.466399 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:51.466375    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4c85361-fb0b-49e4-8f86-3fb92a60ab61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4c85361-fb0b-49e4-8f86-3fb92a60ab61-maas-api-xks-ca\") pod \"d4c85361-fb0b-49e4-8f86-3fb92a60ab61\" (UID: \"d4c85361-fb0b-49e4-8f86-3fb92a60ab61\") "
Oct 30 19:30:51.466611 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:51.466516    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4c85361-fb0b-49e4-8f86-3fb92a60ab61-kube-api-access-swfmq\" (UniqueName: \"kubernetes.io/projected/d4c85361-fb0b-49e4-8f86-3fb92a60ab61-kube-api-access-swfmq\") pod \"d4c85361-fb0b-49e4-8f86-3fb92a60ab61\" (UID: \"d4c85361-fb0b-49e4-8f86-3fb92a60ab61\") "
Oct 30 19:30:51.466702 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:51.466674    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4c85361-fb0b-49e4-8f86-3fb92a60ab61-maas-api-openshift-ca" pod "d4c85361-fb0b-49e4-8f86-3fb92a60ab61" (UID: "d4c85361-fb0b-49e4-8f86-3fb92a60ab61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:51.466781 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:51.466700    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a377cda-465d-45ca-bdd3-5bb94c19ed24-maas-api-openshift-ca" pod "1a377cda-465d-45ca-bdd3-5bb94c19ed24" (UID: "1a377cda-465d-45ca-bdd3-5bb94c19ed24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:51.466781 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:51.466692    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a377cda-465d-45ca-bdd3-5bb94c19ed24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a377cda-465d-45ca-bdd3-5bb94c19ed24-maas-api-xks-ca\") pod \"1a377cda-465d-45ca-bdd3-5bb94c19ed24\" (UID: \"1a377cda-465d-45ca-bdd3-5bb94c19ed24\") "
Oct 30 19:30:51.467700 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:51.467672    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4c85361-fb0b-49e4-8f86-3fb92a60ab61-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:51.467838 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:51.467705    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a377cda-465d-45ca-bdd3-5bb94c19ed24-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:51.469649 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:51.469006    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4c85361-fb0b-49e4-8f86-3fb92a60ab61-maas-api-xks-ca" pod "d4c85361-fb0b-49e4-8f86-3fb92a60ab61" (UID: "d4c85361-fb0b-49e4-8f86-3fb92a60ab61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:30:51.469649 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:51.469051    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a377cda-465d-45ca-bdd3-5bb94c19ed24-kube-api-access-2cgxc" pod "1a377cda-465d-45ca-bdd3-5bb94c19ed24" (UID: "1a377cda-465d-45ca-bdd3-5bb94c19ed24"). InnerVolumeSpecName "kube-api-access-2cgxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:51.469649 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:51.469298    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a377cda-465d-45ca-bdd3-5bb94c19ed24-maas-api-xks-ca" pod "1a377cda-465d-45ca-bdd3-5bb94c19ed24" (UID: "1a377cda-465d-45ca-bdd3-5bb94c19ed24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:30:51.469649 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:51.469411    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4c85361-fb0b-49e4-8f86-3fb92a60ab61-kube-api-access-swfmq" pod "d4c85361-fb0b-49e4-8f86-3fb92a60ab61" (UID: "d4c85361-fb0b-49e4-8f86-3fb92a60ab61"). InnerVolumeSpecName "kube-api-access-swfmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:51.582071 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:51.580808    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4c85361-fb0b-49e4-8f86-3fb92a60ab61-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:51.587646 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:51.587279    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swfmq\" (UniqueName: \"kubernetes.io/projected/d4c85361-fb0b-49e4-8f86-3fb92a60ab61-kube-api-access-swfmq\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:51.587646 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:51.587369    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a377cda-465d-45ca-bdd3-5bb94c19ed24-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:51.587646 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:51.587387    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cgxc\" (UniqueName: \"kubernetes.io/projected/1a377cda-465d-45ca-bdd3-5bb94c19ed24-kube-api-access-2cgxc\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:51.803239 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:51.803147    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b25eed1-4dkn6"
Oct 30 19:30:51.803239 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:51.803166    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b25eed1-4dkn6" event={"ID":"1a377cda-465d-45ca-bdd3-5bb94c19ed24","Type":"ContainerDied","Data":"1472b8f76a464e52b4f7810aeb9fca392c8b1975090ee847e2f1cdf324933d9c"}
Oct 30 19:30:51.803239 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:51.803219    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1472b8f76a464e52b4f7810aeb9fca392c8b1975090ee847e2f1cdf324933d9c"
Oct 30 19:30:51.805088 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:51.805050    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5166f1e7-j5jqp"
Oct 30 19:30:51.805237 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:51.805081    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5166f1e7-j5jqp" event={"ID":"d4c85361-fb0b-49e4-8f86-3fb92a60ab61","Type":"ContainerDied","Data":"dfce87515da3b0b1b72100aeb662ba2ae7ac7d9cacbd7bc443bb28943cdfe4a0"}
Oct 30 19:30:51.805237 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:51.805127    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfce87515da3b0b1b72100aeb662ba2ae7ac7d9cacbd7bc443bb28943cdfe4a0"
Oct 30 19:30:51.979258 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:51.979234    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91cb764-knbd8"
Oct 30 19:30:52.095028 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:52.094935    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1c894c4-4433-4736-b57e-b6bf5db5e698-kube-api-access-fnm8f\" (UniqueName: \"kubernetes.io/projected/a1c894c4-4433-4736-b57e-b6bf5db5e698-kube-api-access-fnm8f\") pod \"a1c894c4-4433-4736-b57e-b6bf5db5e698\" (UID: \"a1c894c4-4433-4736-b57e-b6bf5db5e698\") "
Oct 30 19:30:52.095028 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:52.094991    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1c894c4-4433-4736-b57e-b6bf5db5e698-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1c894c4-4433-4736-b57e-b6bf5db5e698-maas-api-openshift-ca\") pod \"a1c894c4-4433-4736-b57e-b6bf5db5e698\" (UID: \"a1c894c4-4433-4736-b57e-b6bf5db5e698\") "
Oct 30 19:30:52.095269 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:52.095045    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1c894c4-4433-4736-b57e-b6bf5db5e698-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1c894c4-4433-4736-b57e-b6bf5db5e698-maas-api-xks-ca\") pod \"a1c894c4-4433-4736-b57e-b6bf5db5e698\" (UID: \"a1c894c4-4433-4736-b57e-b6bf5db5e698\") "
Oct 30 19:30:52.095399 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:52.095370    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1c894c4-4433-4736-b57e-b6bf5db5e698-maas-api-openshift-ca" pod "a1c894c4-4433-4736-b57e-b6bf5db5e698" (UID: "a1c894c4-4433-4736-b57e-b6bf5db5e698"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:52.096165 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:52.096138    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1c894c4-4433-4736-b57e-b6bf5db5e698-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:52.097777 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:52.097727    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c894c4-4433-4736-b57e-b6bf5db5e698-kube-api-access-fnm8f" pod "a1c894c4-4433-4736-b57e-b6bf5db5e698" (UID: "a1c894c4-4433-4736-b57e-b6bf5db5e698"). InnerVolumeSpecName "kube-api-access-fnm8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:52.097918 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:52.097900    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1c894c4-4433-4736-b57e-b6bf5db5e698-maas-api-xks-ca" pod "a1c894c4-4433-4736-b57e-b6bf5db5e698" (UID: "a1c894c4-4433-4736-b57e-b6bf5db5e698"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:30:52.197356 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:52.197323    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1c894c4-4433-4736-b57e-b6bf5db5e698-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:52.197356 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:52.197354    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnm8f\" (UniqueName: \"kubernetes.io/projected/a1c894c4-4433-4736-b57e-b6bf5db5e698-kube-api-access-fnm8f\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:52.825507 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:52.825468    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91cb764-knbd8" event={"ID":"a1c894c4-4433-4736-b57e-b6bf5db5e698","Type":"ContainerDied","Data":"119f3c4b7612cc10ec8d0816e0afdffbfa17209abd569a95d4589c5b0832ccaa"}
Oct 30 19:30:52.825976 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:52.825529    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="119f3c4b7612cc10ec8d0816e0afdffbfa17209abd569a95d4589c5b0832ccaa"
Oct 30 19:30:52.825976 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:52.825486    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c91cb764-knbd8"
Oct 30 19:30:54.991128 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:30:54.991089    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:30:56.781223 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:56.781187    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0877e187-h82kf"]
Oct 30 19:30:56.781880 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:56.781865    5043 cpu_manager.go:414] "containerMap: removing container" podUID="d4c85361-fb0b-49e4-8f86-3fb92a60ab61" containerName="revoke-keys"
Oct 30 19:30:56.781937 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:56.781880    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4c85361-fb0b-49e4-8f86-3fb92a60ab61" containerName="revoke-keys"
Oct 30 19:30:56.781937 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:56.781917    5043 cpu_manager.go:414] "containerMap: removing container" podUID="1a377cda-465d-45ca-bdd3-5bb94c19ed24" containerName="revoke-keys"
Oct 30 19:30:56.781937 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:56.781922    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a377cda-465d-45ca-bdd3-5bb94c19ed24" containerName="revoke-keys"
Oct 30 19:30:56.782036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:56.781951    5043 cpu_manager.go:414] "containerMap: removing container" podUID="a1c894c4-4433-4736-b57e-b6bf5db5e698" containerName="revoke-keys"
Oct 30 19:30:56.782036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:56.781957    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1c894c4-4433-4736-b57e-b6bf5db5e698" containerName="revoke-keys"
Oct 30 19:30:56.782097 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:56.782054    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1c894c4-4433-4736-b57e-b6bf5db5e698" containerName="revoke-keys"
Oct 30 19:30:56.782097 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:56.782065    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4c85361-fb0b-49e4-8f86-3fb92a60ab61" containerName="revoke-keys"
Oct 30 19:30:56.782097 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:56.782075    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a377cda-465d-45ca-bdd3-5bb94c19ed24" containerName="revoke-keys"
Oct 30 19:30:56.786564 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:56.786534    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0877e187-h82kf"
Oct 30 19:30:56.790456 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:56.790172    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:30:56.790592 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:56.790479    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:30:56.790934 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:56.790916    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:30:56.791313 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:56.791067    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:30:56.794695 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:56.794671    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0877e187-h82kf"]
Oct 30 19:30:56.919670 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:56.919635    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d8f8137-26e5-4a07-8dd4-1155cdd6d896-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0877e187-h82kf\" (UID: \"7d8f8137-26e5-4a07-8dd4-1155cdd6d896\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0877e187-h82kf"
Oct 30 19:30:56.919884 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:56.919864    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d8f8137-26e5-4a07-8dd4-1155cdd6d896-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0877e187-h82kf\" (UID: \"7d8f8137-26e5-4a07-8dd4-1155cdd6d896\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0877e187-h82kf"
Oct 30 19:30:56.919948 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:56.919918    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrntg\" (UniqueName: \"kubernetes.io/projected/7d8f8137-26e5-4a07-8dd4-1155cdd6d896-kube-api-access-mrntg\") pod \"maas-api-revoke-sub-0877e187-h82kf\" (UID: \"7d8f8137-26e5-4a07-8dd4-1155cdd6d896\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0877e187-h82kf"
Oct 30 19:30:57.021584 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:57.021555    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d8f8137-26e5-4a07-8dd4-1155cdd6d896-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0877e187-h82kf\" (UID: \"7d8f8137-26e5-4a07-8dd4-1155cdd6d896\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0877e187-h82kf"
Oct 30 19:30:57.021814 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:57.021711    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d8f8137-26e5-4a07-8dd4-1155cdd6d896-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0877e187-h82kf\" (UID: \"7d8f8137-26e5-4a07-8dd4-1155cdd6d896\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0877e187-h82kf"
Oct 30 19:30:57.021814 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:57.021784    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrntg\" (UniqueName: \"kubernetes.io/projected/7d8f8137-26e5-4a07-8dd4-1155cdd6d896-kube-api-access-mrntg\") pod \"maas-api-revoke-sub-0877e187-h82kf\" (UID: \"7d8f8137-26e5-4a07-8dd4-1155cdd6d896\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0877e187-h82kf"
Oct 30 19:30:57.022410 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:57.022390    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d8f8137-26e5-4a07-8dd4-1155cdd6d896-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0877e187-h82kf\" (UID: \"7d8f8137-26e5-4a07-8dd4-1155cdd6d896\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0877e187-h82kf"
Oct 30 19:30:57.024327 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:57.024302    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d8f8137-26e5-4a07-8dd4-1155cdd6d896-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0877e187-h82kf\" (UID: \"7d8f8137-26e5-4a07-8dd4-1155cdd6d896\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0877e187-h82kf"
Oct 30 19:30:57.030394 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:57.030362    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrntg\" (UniqueName: \"kubernetes.io/projected/7d8f8137-26e5-4a07-8dd4-1155cdd6d896-kube-api-access-mrntg\") pod \"maas-api-revoke-sub-0877e187-h82kf\" (UID: \"7d8f8137-26e5-4a07-8dd4-1155cdd6d896\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0877e187-h82kf"
Oct 30 19:30:57.119984 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:57.119897    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0877e187-h82kf"
Oct 30 19:30:57.424554 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:30:57.424517    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d8f8137_26e5_4a07_8dd4_1155cdd6d896.slice/crio-9f8f92c95c479da76e6771941ab5004da6a31767f560c004c02e395f4b4a4544 WatchSource:0}: Error finding container 9f8f92c95c479da76e6771941ab5004da6a31767f560c004c02e395f4b4a4544: Status 404 returned error can't find the container with id 9f8f92c95c479da76e6771941ab5004da6a31767f560c004c02e395f4b4a4544
Oct 30 19:30:57.426466 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:57.426444    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0877e187-h82kf"]
Oct 30 19:30:57.935625 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:57.935590    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0877e187-h82kf" event={"ID":"7d8f8137-26e5-4a07-8dd4-1155cdd6d896","Type":"ContainerStarted","Data":"9f8f92c95c479da76e6771941ab5004da6a31767f560c004c02e395f4b4a4544"}
Oct 30 19:30:58.046008 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:58.045955    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57"]
Oct 30 19:30:58.065027 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:58.064988    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57"]
Oct 30 19:30:58.065180 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:58.065108    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57"
Oct 30 19:30:58.107089 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:58.107057    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-5e333f16-gw-openshift-default-dockercfg-9js2k\""
Oct 30 19:30:58.364816 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:58.354036    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8be979e-a5c4-481a-bd2e-e7275f28c83a-istio-data\") pod \"e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57\" (UID: \"e8be979e-a5c4-481a-bd2e-e7275f28c83a\") " pod="openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57"
Oct 30 19:30:58.364816 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:58.354089    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8be979e-a5c4-481a-bd2e-e7275f28c83a-istio-podinfo\") pod \"e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57\" (UID: \"e8be979e-a5c4-481a-bd2e-e7275f28c83a\") " pod="openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57"
Oct 30 19:30:58.364816 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:58.354132    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8be979e-a5c4-481a-bd2e-e7275f28c83a-istiod-ca-cert\") pod \"e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57\" (UID: \"e8be979e-a5c4-481a-bd2e-e7275f28c83a\") " pod="openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57"
Oct 30 19:30:58.364816 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:58.354185    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjpp6\" (UniqueName: \"kubernetes.io/projected/e8be979e-a5c4-481a-bd2e-e7275f28c83a-kube-api-access-fjpp6\") pod \"e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57\" (UID: \"e8be979e-a5c4-481a-bd2e-e7275f28c83a\") " pod="openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57"
Oct 30 19:30:58.364816 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:58.354211    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8be979e-a5c4-481a-bd2e-e7275f28c83a-workload-socket\") pod \"e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57\" (UID: \"e8be979e-a5c4-481a-bd2e-e7275f28c83a\") " pod="openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57"
Oct 30 19:30:58.364816 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:58.354305    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8be979e-a5c4-481a-bd2e-e7275f28c83a-credential-socket\") pod \"e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57\" (UID: \"e8be979e-a5c4-481a-bd2e-e7275f28c83a\") " pod="openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57"
Oct 30 19:30:58.364816 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:58.354387    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8be979e-a5c4-481a-bd2e-e7275f28c83a-workload-certs\") pod \"e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57\" (UID: \"e8be979e-a5c4-481a-bd2e-e7275f28c83a\") " pod="openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57"
Oct 30 19:30:58.364816 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:58.354417    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8be979e-a5c4-481a-bd2e-e7275f28c83a-istio-envoy\") pod \"e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57\" (UID: \"e8be979e-a5c4-481a-bd2e-e7275f28c83a\") " pod="openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57"
Oct 30 19:30:58.373759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:58.365507    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8be979e-a5c4-481a-bd2e-e7275f28c83a-istio-token\") pod \"e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57\" (UID: \"e8be979e-a5c4-481a-bd2e-e7275f28c83a\") " pod="openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57"
Oct 30 19:30:58.477785 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:58.472272    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8be979e-a5c4-481a-bd2e-e7275f28c83a-istio-data\") pod \"e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57\" (UID: \"e8be979e-a5c4-481a-bd2e-e7275f28c83a\") " pod="openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57"
Oct 30 19:30:58.477785 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:58.472316    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8be979e-a5c4-481a-bd2e-e7275f28c83a-istio-podinfo\") pod \"e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57\" (UID: \"e8be979e-a5c4-481a-bd2e-e7275f28c83a\") " pod="openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57"
Oct 30 19:30:58.477785 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:58.472351    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8be979e-a5c4-481a-bd2e-e7275f28c83a-istiod-ca-cert\") pod \"e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57\" (UID: \"e8be979e-a5c4-481a-bd2e-e7275f28c83a\") " pod="openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57"
Oct 30 19:30:58.477785 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:58.472403    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjpp6\" (UniqueName: \"kubernetes.io/projected/e8be979e-a5c4-481a-bd2e-e7275f28c83a-kube-api-access-fjpp6\") pod \"e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57\" (UID: \"e8be979e-a5c4-481a-bd2e-e7275f28c83a\") " pod="openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57"
Oct 30 19:30:58.477785 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:58.472428    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8be979e-a5c4-481a-bd2e-e7275f28c83a-workload-socket\") pod \"e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57\" (UID: \"e8be979e-a5c4-481a-bd2e-e7275f28c83a\") " pod="openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57"
Oct 30 19:30:58.477785 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:58.472479    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8be979e-a5c4-481a-bd2e-e7275f28c83a-credential-socket\") pod \"e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57\" (UID: \"e8be979e-a5c4-481a-bd2e-e7275f28c83a\") " pod="openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57"
Oct 30 19:30:58.477785 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:58.472563    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8be979e-a5c4-481a-bd2e-e7275f28c83a-workload-certs\") pod \"e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57\" (UID: \"e8be979e-a5c4-481a-bd2e-e7275f28c83a\") " pod="openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57"
Oct 30 19:30:58.477785 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:58.472594    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8be979e-a5c4-481a-bd2e-e7275f28c83a-istio-envoy\") pod \"e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57\" (UID: \"e8be979e-a5c4-481a-bd2e-e7275f28c83a\") " pod="openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57"
Oct 30 19:30:58.477785 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:58.472768    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8be979e-a5c4-481a-bd2e-e7275f28c83a-istio-token\") pod \"e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57\" (UID: \"e8be979e-a5c4-481a-bd2e-e7275f28c83a\") " pod="openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57"
Oct 30 19:30:58.477785 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:58.473632    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8be979e-a5c4-481a-bd2e-e7275f28c83a-istio-data\") pod \"e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57\" (UID: \"e8be979e-a5c4-481a-bd2e-e7275f28c83a\") " pod="openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57"
Oct 30 19:30:58.477785 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:58.476100    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8be979e-a5c4-481a-bd2e-e7275f28c83a-workload-socket\") pod \"e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57\" (UID: \"e8be979e-a5c4-481a-bd2e-e7275f28c83a\") " pod="openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57"
Oct 30 19:30:58.477785 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:58.476712    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8be979e-a5c4-481a-bd2e-e7275f28c83a-istiod-ca-cert\") pod \"e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57\" (UID: \"e8be979e-a5c4-481a-bd2e-e7275f28c83a\") " pod="openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57"
Oct 30 19:30:58.487797 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:58.483976    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8be979e-a5c4-481a-bd2e-e7275f28c83a-workload-certs\") pod \"e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57\" (UID: \"e8be979e-a5c4-481a-bd2e-e7275f28c83a\") " pod="openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57"
Oct 30 19:30:58.487797 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:58.484522    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8be979e-a5c4-481a-bd2e-e7275f28c83a-credential-socket\") pod \"e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57\" (UID: \"e8be979e-a5c4-481a-bd2e-e7275f28c83a\") " pod="openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57"
Oct 30 19:30:58.494350 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:58.494315    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8be979e-a5c4-481a-bd2e-e7275f28c83a-istio-envoy\") pod \"e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57\" (UID: \"e8be979e-a5c4-481a-bd2e-e7275f28c83a\") " pod="openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57"
Oct 30 19:30:58.496388 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:58.496355    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8be979e-a5c4-481a-bd2e-e7275f28c83a-istio-podinfo\") pod \"e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57\" (UID: \"e8be979e-a5c4-481a-bd2e-e7275f28c83a\") " pod="openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57"
Oct 30 19:30:58.500138 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:58.500107    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8be979e-a5c4-481a-bd2e-e7275f28c83a-istio-token\") pod \"e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57\" (UID: \"e8be979e-a5c4-481a-bd2e-e7275f28c83a\") " pod="openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57"
Oct 30 19:30:58.501577 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:58.501552    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjpp6\" (UniqueName: \"kubernetes.io/projected/e8be979e-a5c4-481a-bd2e-e7275f28c83a-kube-api-access-fjpp6\") pod \"e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57\" (UID: \"e8be979e-a5c4-481a-bd2e-e7275f28c83a\") " pod="openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57"
Oct 30 19:30:58.702157 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:58.702125    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57"
Oct 30 19:30:58.946266 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:58.946179    5043 generic.go:356] "Generic (PLEG): container finished" podID="7d8f8137-26e5-4a07-8dd4-1155cdd6d896" containerID="40be87fb3f49609b445d98f54a95d3a782c0246dabf4bd81555f347cf7c08cbd" exitCode=0
Oct 30 19:30:58.946644 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:58.946269    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0877e187-h82kf" event={"ID":"7d8f8137-26e5-4a07-8dd4-1155cdd6d896","Type":"ContainerDied","Data":"40be87fb3f49609b445d98f54a95d3a782c0246dabf4bd81555f347cf7c08cbd"}
Oct 30 19:30:59.004091 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:59.004060    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10db525e-hsnq6"]
Oct 30 19:30:59.010253 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:59.009976    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10db525e-hsnq6"
Oct 30 19:30:59.022517 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:59.021184    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10db525e-hsnq6"]
Oct 30 19:30:59.028759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:59.028716    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57"]
Oct 30 19:30:59.036932 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:59.036895    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:30:59.037036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:59.036994    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:30:59.037105 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:59.037036    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:30:59.092858 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:59.092832    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6578d\" (UniqueName: \"kubernetes.io/projected/f0590df0-bd23-418f-a4d1-e84f723de4bf-kube-api-access-6578d\") pod \"maas-api-revoke-sub-10db525e-hsnq6\" (UID: \"f0590df0-bd23-418f-a4d1-e84f723de4bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10db525e-hsnq6"
Oct 30 19:30:59.093113 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:59.093092    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0590df0-bd23-418f-a4d1-e84f723de4bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10db525e-hsnq6\" (UID: \"f0590df0-bd23-418f-a4d1-e84f723de4bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10db525e-hsnq6"
Oct 30 19:30:59.093215 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:59.093165    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0590df0-bd23-418f-a4d1-e84f723de4bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10db525e-hsnq6\" (UID: \"f0590df0-bd23-418f-a4d1-e84f723de4bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10db525e-hsnq6"
Oct 30 19:30:59.195539 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:59.195490    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6578d\" (UniqueName: \"kubernetes.io/projected/f0590df0-bd23-418f-a4d1-e84f723de4bf-kube-api-access-6578d\") pod \"maas-api-revoke-sub-10db525e-hsnq6\" (UID: \"f0590df0-bd23-418f-a4d1-e84f723de4bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10db525e-hsnq6"
Oct 30 19:30:59.195762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:59.195729    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0590df0-bd23-418f-a4d1-e84f723de4bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10db525e-hsnq6\" (UID: \"f0590df0-bd23-418f-a4d1-e84f723de4bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10db525e-hsnq6"
Oct 30 19:30:59.195846 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:59.195824    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0590df0-bd23-418f-a4d1-e84f723de4bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10db525e-hsnq6\" (UID: \"f0590df0-bd23-418f-a4d1-e84f723de4bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10db525e-hsnq6"
Oct 30 19:30:59.196712 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:59.196646    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0590df0-bd23-418f-a4d1-e84f723de4bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10db525e-hsnq6\" (UID: \"f0590df0-bd23-418f-a4d1-e84f723de4bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10db525e-hsnq6"
Oct 30 19:30:59.198932 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:59.198898    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0590df0-bd23-418f-a4d1-e84f723de4bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10db525e-hsnq6\" (UID: \"f0590df0-bd23-418f-a4d1-e84f723de4bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10db525e-hsnq6"
Oct 30 19:30:59.204875 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:59.204851    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6578d\" (UniqueName: \"kubernetes.io/projected/f0590df0-bd23-418f-a4d1-e84f723de4bf-kube-api-access-6578d\") pod \"maas-api-revoke-sub-10db525e-hsnq6\" (UID: \"f0590df0-bd23-418f-a4d1-e84f723de4bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10db525e-hsnq6"
Oct 30 19:30:59.368578 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:59.368264    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10db525e-hsnq6"
Oct 30 19:30:59.725726 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:30:59.722840    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0590df0_bd23_418f_a4d1_e84f723de4bf.slice/crio-8d93aaa348b8caee0182bf91cdaa0d0505a5a36029316c1b98dbfbbc195a5ddb WatchSource:0}: Error finding container 8d93aaa348b8caee0182bf91cdaa0d0505a5a36029316c1b98dbfbbc195a5ddb: Status 404 returned error can't find the container with id 8d93aaa348b8caee0182bf91cdaa0d0505a5a36029316c1b98dbfbbc195a5ddb
Oct 30 19:30:59.725726 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:59.725601    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10db525e-hsnq6"]
Oct 30 19:30:59.783009 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:59.782983    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-2ae6628f-54ff895c9d-4d9pk"
Oct 30 19:30:59.962130 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:59.962097    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10db525e-hsnq6" event={"ID":"f0590df0-bd23-418f-a4d1-e84f723de4bf","Type":"ContainerStarted","Data":"8d93aaa348b8caee0182bf91cdaa0d0505a5a36029316c1b98dbfbbc195a5ddb"}
Oct 30 19:30:59.964212 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:59.964184    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57" event={"ID":"e8be979e-a5c4-481a-bd2e-e7275f28c83a","Type":"ContainerStarted","Data":"734b07a83502f49f5402279d544a17dd8e3a503eae4e696b948c0692b817f2df"}
Oct 30 19:30:59.964375 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:59.964233    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57" event={"ID":"e8be979e-a5c4-481a-bd2e-e7275f28c83a","Type":"ContainerStarted","Data":"a5be4325bd1791999046816d93e5956606759de11fe2f9135f4cb04a1f0b078d"}
Oct 30 19:30:59.996172 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:30:59.996064    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57" podStartSLOduration=1.99604512 podStartE2EDuration="1.99604512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:59.985047534 +0000 UTC m=+879.496107492" watchObservedRunningTime="2026-10-30 19:30:59.99604512 +0000 UTC m=+879.507105085"
Oct 30 19:31:00.304755 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:00.304717    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0877e187-h82kf"
Oct 30 19:31:00.385261 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:00.385230    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d8f8137-26e5-4a07-8dd4-1155cdd6d896-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d8f8137-26e5-4a07-8dd4-1155cdd6d896-maas-api-xks-ca\") pod \"7d8f8137-26e5-4a07-8dd4-1155cdd6d896\" (UID: \"7d8f8137-26e5-4a07-8dd4-1155cdd6d896\") "
Oct 30 19:31:00.385470 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:00.385311    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d8f8137-26e5-4a07-8dd4-1155cdd6d896-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d8f8137-26e5-4a07-8dd4-1155cdd6d896-maas-api-openshift-ca\") pod \"7d8f8137-26e5-4a07-8dd4-1155cdd6d896\" (UID: \"7d8f8137-26e5-4a07-8dd4-1155cdd6d896\") "
Oct 30 19:31:00.385470 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:00.385385    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d8f8137-26e5-4a07-8dd4-1155cdd6d896-kube-api-access-mrntg\" (UniqueName: \"kubernetes.io/projected/7d8f8137-26e5-4a07-8dd4-1155cdd6d896-kube-api-access-mrntg\") pod \"7d8f8137-26e5-4a07-8dd4-1155cdd6d896\" (UID: \"7d8f8137-26e5-4a07-8dd4-1155cdd6d896\") "
Oct 30 19:31:00.386413 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:00.386379    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d8f8137-26e5-4a07-8dd4-1155cdd6d896-maas-api-openshift-ca" pod "7d8f8137-26e5-4a07-8dd4-1155cdd6d896" (UID: "7d8f8137-26e5-4a07-8dd4-1155cdd6d896"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:31:00.388334 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:00.388308    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d8f8137-26e5-4a07-8dd4-1155cdd6d896-maas-api-xks-ca" pod "7d8f8137-26e5-4a07-8dd4-1155cdd6d896" (UID: "7d8f8137-26e5-4a07-8dd4-1155cdd6d896"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:31:00.388334 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:00.388320    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d8f8137-26e5-4a07-8dd4-1155cdd6d896-kube-api-access-mrntg" pod "7d8f8137-26e5-4a07-8dd4-1155cdd6d896" (UID: "7d8f8137-26e5-4a07-8dd4-1155cdd6d896"). InnerVolumeSpecName "kube-api-access-mrntg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:31:00.487638 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:00.487606    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrntg\" (UniqueName: \"kubernetes.io/projected/7d8f8137-26e5-4a07-8dd4-1155cdd6d896-kube-api-access-mrntg\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:00.487638 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:00.487633    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d8f8137-26e5-4a07-8dd4-1155cdd6d896-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:00.487638 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:00.487644    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d8f8137-26e5-4a07-8dd4-1155cdd6d896-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:00.703242 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:00.703212    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57"
Oct 30 19:31:00.708811 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:00.708789    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57"
Oct 30 19:31:00.999061 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:00.997007    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0877e187-h82kf" event={"ID":"7d8f8137-26e5-4a07-8dd4-1155cdd6d896","Type":"ContainerDied","Data":"9f8f92c95c479da76e6771941ab5004da6a31767f560c004c02e395f4b4a4544"}
Oct 30 19:31:00.999061 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:00.997067    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f8f92c95c479da76e6771941ab5004da6a31767f560c004c02e395f4b4a4544"
Oct 30 19:31:00.999061 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:00.997135    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0877e187-h82kf"
Oct 30 19:31:01.009757 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:01.009145    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10db525e-hsnq6" event={"ID":"f0590df0-bd23-418f-a4d1-e84f723de4bf","Type":"ContainerStarted","Data":"90580a5dacd0857d36bb4f7271e5e3a06bde42cb33d5e2a47c2f5fe0093155b1"}
Oct 30 19:31:01.009757 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:01.009206    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57"
Oct 30 19:31:01.019081 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:01.019051    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57"
Oct 30 19:31:02.022270 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:02.022232    5043 generic.go:356] "Generic (PLEG): container finished" podID="f0590df0-bd23-418f-a4d1-e84f723de4bf" containerID="90580a5dacd0857d36bb4f7271e5e3a06bde42cb33d5e2a47c2f5fe0093155b1" exitCode=0
Oct 30 19:31:02.022724 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:02.022340    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10db525e-hsnq6" event={"ID":"f0590df0-bd23-418f-a4d1-e84f723de4bf","Type":"ContainerDied","Data":"90580a5dacd0857d36bb4f7271e5e3a06bde42cb33d5e2a47c2f5fe0093155b1"}
Oct 30 19:31:02.280673 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:02.280634    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5e333f16-f5f6bd6dc-jfs9n"]
Oct 30 19:31:02.282619 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:02.282491    5043 cpu_manager.go:414] "containerMap: removing container" podUID="7d8f8137-26e5-4a07-8dd4-1155cdd6d896" containerName="revoke-keys"
Oct 30 19:31:02.282619 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:02.282512    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d8f8137-26e5-4a07-8dd4-1155cdd6d896" containerName="revoke-keys"
Oct 30 19:31:02.282781 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:02.282757    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d8f8137-26e5-4a07-8dd4-1155cdd6d896" containerName="revoke-keys"
Oct 30 19:31:02.290299 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:02.290275    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5e333f16-f5f6bd6dc-jfs9n"
Oct 30 19:31:02.299143 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:02.297237    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-5e333f16\""
Oct 30 19:31:02.299143 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:02.298467    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-5e333f16-dockercfg-ckxst\""
Oct 30 19:31:02.306327 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:02.306308    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5e333f16-68c977c4bd-rhgmr"]
Oct 30 19:31:02.315267 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:02.315245    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5e333f16-f5f6bd6dc-jfs9n"]
Oct 30 19:31:02.315439 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:02.315349    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-5e333f16-68c977c4bd-rhgmr"
Oct 30 19:31:02.315716 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:02.315694    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10db525e-hsnq6"
Oct 30 19:31:02.321495 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:02.321475    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5e333f16-68c977c4bd-rhgmr"]
Oct 30 19:31:02.385760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:02.379867    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rzff\" (UniqueName: \"kubernetes.io/projected/c555e0a2-316b-419d-9b37-80c44bbe9cfc-kube-api-access-9rzff\") pod \"payload-pre-processing-e2e-migrate-5e333f16-f5f6bd6dc-jfs9n\" (UID: \"c555e0a2-316b-419d-9b37-80c44bbe9cfc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5e333f16-f5f6bd6dc-jfs9n"
Oct 30 19:31:02.385760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:02.380132    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c555e0a2-316b-419d-9b37-80c44bbe9cfc-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-5e333f16-f5f6bd6dc-jfs9n\" (UID: \"c555e0a2-316b-419d-9b37-80c44bbe9cfc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5e333f16-f5f6bd6dc-jfs9n"
Oct 30 19:31:02.633058 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:02.631777    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c555e0a2-316b-419d-9b37-80c44bbe9cfc-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-5e333f16-f5f6bd6dc-jfs9n\" (UID: \"c555e0a2-316b-419d-9b37-80c44bbe9cfc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5e333f16-f5f6bd6dc-jfs9n"
Oct 30 19:31:02.633904 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:02.633525    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rzff\" (UniqueName: \"kubernetes.io/projected/c555e0a2-316b-419d-9b37-80c44bbe9cfc-kube-api-access-9rzff\") pod \"payload-pre-processing-e2e-migrate-5e333f16-f5f6bd6dc-jfs9n\" (UID: \"c555e0a2-316b-419d-9b37-80c44bbe9cfc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5e333f16-f5f6bd6dc-jfs9n"
Oct 30 19:31:02.635643 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:02.635149    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c555e0a2-316b-419d-9b37-80c44bbe9cfc-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-5e333f16-f5f6bd6dc-jfs9n\" (UID: \"c555e0a2-316b-419d-9b37-80c44bbe9cfc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5e333f16-f5f6bd6dc-jfs9n"
Oct 30 19:31:02.677436 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:02.677234    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rzff\" (UniqueName: \"kubernetes.io/projected/c555e0a2-316b-419d-9b37-80c44bbe9cfc-kube-api-access-9rzff\") pod \"payload-pre-processing-e2e-migrate-5e333f16-f5f6bd6dc-jfs9n\" (UID: \"c555e0a2-316b-419d-9b37-80c44bbe9cfc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5e333f16-f5f6bd6dc-jfs9n"
Oct 30 19:31:02.735168 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:02.735120    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0590df0-bd23-418f-a4d1-e84f723de4bf-kube-api-access-6578d\" (UniqueName: \"kubernetes.io/projected/f0590df0-bd23-418f-a4d1-e84f723de4bf-kube-api-access-6578d\") pod \"f0590df0-bd23-418f-a4d1-e84f723de4bf\" (UID: \"f0590df0-bd23-418f-a4d1-e84f723de4bf\") "
Oct 30 19:31:02.735348 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:02.735255    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0590df0-bd23-418f-a4d1-e84f723de4bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0590df0-bd23-418f-a4d1-e84f723de4bf-maas-api-xks-ca\") pod \"f0590df0-bd23-418f-a4d1-e84f723de4bf\" (UID: \"f0590df0-bd23-418f-a4d1-e84f723de4bf\") "
Oct 30 19:31:02.735348 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:02.735335    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0590df0-bd23-418f-a4d1-e84f723de4bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0590df0-bd23-418f-a4d1-e84f723de4bf-maas-api-openshift-ca\") pod \"f0590df0-bd23-418f-a4d1-e84f723de4bf\" (UID: \"f0590df0-bd23-418f-a4d1-e84f723de4bf\") "
Oct 30 19:31:02.735844 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:02.735813    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0590df0-bd23-418f-a4d1-e84f723de4bf-maas-api-openshift-ca" pod "f0590df0-bd23-418f-a4d1-e84f723de4bf" (UID: "f0590df0-bd23-418f-a4d1-e84f723de4bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:31:02.736275 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:02.736244    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vspmb\" (UniqueName: \"kubernetes.io/projected/0669101d-0bf5-4c6c-a088-180749a12376-kube-api-access-vspmb\") pod \"payload-processing-e2e-migrate-5e333f16-68c977c4bd-rhgmr\" (UID: \"0669101d-0bf5-4c6c-a088-180749a12376\") " pod="openshift-ingress/payload-processing-e2e-migrate-5e333f16-68c977c4bd-rhgmr"
Oct 30 19:31:02.736453 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:02.736404    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0669101d-0bf5-4c6c-a088-180749a12376-plugins-config-volume\") pod \"payload-processing-e2e-migrate-5e333f16-68c977c4bd-rhgmr\" (UID: \"0669101d-0bf5-4c6c-a088-180749a12376\") " pod="openshift-ingress/payload-processing-e2e-migrate-5e333f16-68c977c4bd-rhgmr"
Oct 30 19:31:02.736887 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:02.736849    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0590df0-bd23-418f-a4d1-e84f723de4bf-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:02.737885 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:02.737852    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0590df0-bd23-418f-a4d1-e84f723de4bf-maas-api-xks-ca" pod "f0590df0-bd23-418f-a4d1-e84f723de4bf" (UID: "f0590df0-bd23-418f-a4d1-e84f723de4bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:31:02.738241 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:02.738218    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0590df0-bd23-418f-a4d1-e84f723de4bf-kube-api-access-6578d" pod "f0590df0-bd23-418f-a4d1-e84f723de4bf" (UID: "f0590df0-bd23-418f-a4d1-e84f723de4bf"). InnerVolumeSpecName "kube-api-access-6578d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:31:02.837918 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:02.837887    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vspmb\" (UniqueName: \"kubernetes.io/projected/0669101d-0bf5-4c6c-a088-180749a12376-kube-api-access-vspmb\") pod \"payload-processing-e2e-migrate-5e333f16-68c977c4bd-rhgmr\" (UID: \"0669101d-0bf5-4c6c-a088-180749a12376\") " pod="openshift-ingress/payload-processing-e2e-migrate-5e333f16-68c977c4bd-rhgmr"
Oct 30 19:31:02.838119 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:02.837984    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0669101d-0bf5-4c6c-a088-180749a12376-plugins-config-volume\") pod \"payload-processing-e2e-migrate-5e333f16-68c977c4bd-rhgmr\" (UID: \"0669101d-0bf5-4c6c-a088-180749a12376\") " pod="openshift-ingress/payload-processing-e2e-migrate-5e333f16-68c977c4bd-rhgmr"
Oct 30 19:31:02.838485 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:02.838462    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6578d\" (UniqueName: \"kubernetes.io/projected/f0590df0-bd23-418f-a4d1-e84f723de4bf-kube-api-access-6578d\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:02.838485 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:02.838483    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0590df0-bd23-418f-a4d1-e84f723de4bf-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:02.838663 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:02.838642    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0669101d-0bf5-4c6c-a088-180749a12376-plugins-config-volume\") pod \"payload-processing-e2e-migrate-5e333f16-68c977c4bd-rhgmr\" (UID: \"0669101d-0bf5-4c6c-a088-180749a12376\") " pod="openshift-ingress/payload-processing-e2e-migrate-5e333f16-68c977c4bd-rhgmr"
Oct 30 19:31:02.850257 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:02.850236    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vspmb\" (UniqueName: \"kubernetes.io/projected/0669101d-0bf5-4c6c-a088-180749a12376-kube-api-access-vspmb\") pod \"payload-processing-e2e-migrate-5e333f16-68c977c4bd-rhgmr\" (UID: \"0669101d-0bf5-4c6c-a088-180749a12376\") " pod="openshift-ingress/payload-processing-e2e-migrate-5e333f16-68c977c4bd-rhgmr"
Oct 30 19:31:02.927405 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:02.927322    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5e333f16-f5f6bd6dc-jfs9n"
Oct 30 19:31:02.936859 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:02.936834    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-5e333f16-68c977c4bd-rhgmr"
Oct 30 19:31:03.049206 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:03.049184    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10db525e-hsnq6"
Oct 30 19:31:03.049539 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:03.049217    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10db525e-hsnq6" event={"ID":"f0590df0-bd23-418f-a4d1-e84f723de4bf","Type":"ContainerDied","Data":"8d93aaa348b8caee0182bf91cdaa0d0505a5a36029316c1b98dbfbbc195a5ddb"}
Oct 30 19:31:03.049539 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:03.049268    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d93aaa348b8caee0182bf91cdaa0d0505a5a36029316c1b98dbfbbc195a5ddb"
Oct 30 19:31:03.239895 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:03.239870    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5e333f16-f5f6bd6dc-jfs9n"]
Oct 30 19:31:03.247441 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:03.247411    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5e333f16-68c977c4bd-rhgmr"]
Oct 30 19:31:04.060985 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:04.060954    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-5e333f16-68c977c4bd-rhgmr" event={"ID":"0669101d-0bf5-4c6c-a088-180749a12376","Type":"ContainerStarted","Data":"723f67b7845b5f44754fa62fa12c0f32f810de33f8d63db56f316d39d4329578"}
Oct 30 19:31:04.061464 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:04.061006    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-5e333f16-68c977c4bd-rhgmr" event={"ID":"0669101d-0bf5-4c6c-a088-180749a12376","Type":"ContainerStarted","Data":"810bf71e3736846a362fd0c4dcc47f27eb92bcee8c89c67fbe3239980d6da830"}
Oct 30 19:31:04.061464 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:04.061060    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-5e333f16-68c977c4bd-rhgmr"
Oct 30 19:31:04.063060 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:04.063032    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5e333f16-f5f6bd6dc-jfs9n" event={"ID":"c555e0a2-316b-419d-9b37-80c44bbe9cfc","Type":"ContainerStarted","Data":"383d89ee213e288eafa68e0cdc577b8abd6391d40d5446d9fb0580a637138f77"}
Oct 30 19:31:04.063203 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:04.063084    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5e333f16-f5f6bd6dc-jfs9n" event={"ID":"c555e0a2-316b-419d-9b37-80c44bbe9cfc","Type":"ContainerStarted","Data":"8f872056044465f71f94a07a72dd95c337ca6e5ce44ab13752a55b54710cbdd1"}
Oct 30 19:31:04.063203 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:04.063125    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5e333f16-f5f6bd6dc-jfs9n"
Oct 30 19:31:04.088285 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:04.088215    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-5e333f16-68c977c4bd-rhgmr" podStartSLOduration=2.088197021 podStartE2EDuration="2.088197021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:04.087952928 +0000 UTC m=+883.599012891" watchObservedRunningTime="2026-10-30 19:31:04.088197021 +0000 UTC m=+883.599256985"
Oct 30 19:31:04.109835 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:04.109784    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5e333f16-f5f6bd6dc-jfs9n" podStartSLOduration=2.109770399 podStartE2EDuration="2.109770399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:04.105662599 +0000 UTC m=+883.616722563" watchObservedRunningTime="2026-10-30 19:31:04.109770399 +0000 UTC m=+883.620830363"
Oct 30 19:31:05.591762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:05.589920    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77930c58-zfnhc"]
Oct 30 19:31:05.629983 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:05.617613    5043 cpu_manager.go:414] "containerMap: removing container" podUID="f0590df0-bd23-418f-a4d1-e84f723de4bf" containerName="revoke-keys"
Oct 30 19:31:05.629983 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:05.617637    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0590df0-bd23-418f-a4d1-e84f723de4bf" containerName="revoke-keys"
Oct 30 19:31:05.629983 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:05.617873    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0590df0-bd23-418f-a4d1-e84f723de4bf" containerName="revoke-keys"
Oct 30 19:31:05.651903 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:05.651872    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77930c58-zfnhc"]
Oct 30 19:31:05.652280 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:05.652260    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77930c58-zfnhc"
Oct 30 19:31:05.671762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:05.664000    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:31:05.695485 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:05.674096    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:31:05.695485 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:05.675925    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:31:05.695485 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:05.675981    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:31:05.695485 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:05.676007    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:31:05.820765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:05.802137    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz7tg\" (UniqueName: \"kubernetes.io/projected/4088447a-a476-4088-b3f7-40b825b7ac15-kube-api-access-lz7tg\") pod \"maas-api-revoke-keys-e2e-ait-77930c58-zfnhc\" (UID: \"4088447a-a476-4088-b3f7-40b825b7ac15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77930c58-zfnhc"
Oct 30 19:31:05.820765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:05.802313    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4088447a-a476-4088-b3f7-40b825b7ac15-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-77930c58-zfnhc\" (UID: \"4088447a-a476-4088-b3f7-40b825b7ac15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77930c58-zfnhc"
Oct 30 19:31:05.820765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:05.802345    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4088447a-a476-4088-b3f7-40b825b7ac15-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-77930c58-zfnhc\" (UID: \"4088447a-a476-4088-b3f7-40b825b7ac15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77930c58-zfnhc"
Oct 30 19:31:05.834881 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:05.834835    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:05.927808 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:05.915881    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz7tg\" (UniqueName: \"kubernetes.io/projected/4088447a-a476-4088-b3f7-40b825b7ac15-kube-api-access-lz7tg\") pod \"maas-api-revoke-keys-e2e-ait-77930c58-zfnhc\" (UID: \"4088447a-a476-4088-b3f7-40b825b7ac15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77930c58-zfnhc"
Oct 30 19:31:05.927808 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:05.916039    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4088447a-a476-4088-b3f7-40b825b7ac15-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-77930c58-zfnhc\" (UID: \"4088447a-a476-4088-b3f7-40b825b7ac15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77930c58-zfnhc"
Oct 30 19:31:05.927808 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:05.916067    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4088447a-a476-4088-b3f7-40b825b7ac15-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-77930c58-zfnhc\" (UID: \"4088447a-a476-4088-b3f7-40b825b7ac15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77930c58-zfnhc"
Oct 30 19:31:05.927808 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:05.921572    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4088447a-a476-4088-b3f7-40b825b7ac15-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-77930c58-zfnhc\" (UID: \"4088447a-a476-4088-b3f7-40b825b7ac15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77930c58-zfnhc"
Oct 30 19:31:05.927808 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:05.925509    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4088447a-a476-4088-b3f7-40b825b7ac15-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-77930c58-zfnhc\" (UID: \"4088447a-a476-4088-b3f7-40b825b7ac15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77930c58-zfnhc"
Oct 30 19:31:05.955941 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:05.955860    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz7tg\" (UniqueName: \"kubernetes.io/projected/4088447a-a476-4088-b3f7-40b825b7ac15-kube-api-access-lz7tg\") pod \"maas-api-revoke-keys-e2e-ait-77930c58-zfnhc\" (UID: \"4088447a-a476-4088-b3f7-40b825b7ac15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77930c58-zfnhc"
Oct 30 19:31:05.974367 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:05.972855    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:31:05.993565 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:05.991832    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:06.028212 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:06.028171    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77930c58-zfnhc"
Oct 30 19:31:06.209817 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:06.206552    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:06.210088 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:06.210051    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:31:06.214363 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:06.212184    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:06.214363 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:06.212526    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:06.264670 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:06.264622    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:06.326468 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:06.326423    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:06.705813 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:06.704676    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:06.718765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:06.718425    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:31:06.888774 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:06.884411    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:31:07.062766 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:07.057630    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:07.094806 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:07.071776    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:31:07.106916 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:07.095517    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:31:07.261813 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:07.230214    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:07.261813 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:07.235241    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded"
Oct 30 19:31:07.261813 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:07.235989    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:31:07.551390 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:07.550269    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:31:07.605148 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:07.605089    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:31:07.814372 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:07.665918    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:07.814372 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:31:07.718607    5043 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1s"
Oct 30 19:31:07.814372 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:07.718780    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:31:07.814372 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:07.718826    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:31:07.814372 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:07.718864    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:31:07.814372 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:07.718904    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"
Oct 30 19:31:07.814372 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:07.718940    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:31:07.814372 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:07.718976    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:31:07.814372 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:07.719010    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:31:07.814372 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:07.719043    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:31:07.814372 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:07.719074    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"
Oct 30 19:31:07.952825 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:07.712917    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:07.992677 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:07.992632    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:08.064819 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:08.037075    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:08.089758 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:08.057321    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": context deadline exceeded"
Oct 30 19:31:08.090089 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:08.065115    5043 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:31:08.090614 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:08.090584    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf" podUID="4fea9e2a-77c0-48c4-8517-3e104fc043b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:08.091015 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:08.085929    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:08.116959 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:08.116899    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:08.588764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:08.588203    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded"
Oct 30 19:31:08.615762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:08.610474    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:31:08.615762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:08.610868    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:08.619892 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:08.619849    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:31:08.750768 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:08.711796    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:31:08.953765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:08.937108    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 30 19:31:08.991760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:08.974002    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:09.074366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:09.074297    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:09.119872 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:09.114185    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:31:09.140771 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:09.128751    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:31:09.256319 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:09.256270    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:31:09.552457 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:09.552419    5043 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:31:09.587195 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:09.587153    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t" podUID="4897f8f5-579f-45ed-a964-f2379eb52848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:09.588402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:09.588357    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:09.645394 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:09.645338    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:31:09.646213 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:09.646170    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:31:09.646648 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:09.646626    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:31:09.737893 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:09.737823    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:31:09.896922 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:09.896878    5043 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:31:09.897200 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:09.897168    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c" podUID="33a5f4d9-fe3f-40ad-904d-db5ac720c81e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:09.957771 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:09.939527    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 30 19:31:09.970545 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:09.970490    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:09.994571 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:09.994450    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:10.028509 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:10.028475    5043 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:31:10.028839 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:10.028807    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt" podUID="359c47fc-f8af-43f8-865b-1f187fb2672c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:10.123786 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:31:10.116727    5043 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.141s"
Oct 30 19:31:10.125564 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:10.124089    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:10.198275 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:10.197277    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:10.265750 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:10.263860    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:31:10.265750 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:10.265333    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": context deadline exceeded"
Oct 30 19:31:10.374880 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:10.374823    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77930c58-zfnhc"]
Oct 30 19:31:10.609844 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:10.603120    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded"
Oct 30 19:31:10.769419 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:10.662887    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:31:10.769419 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:10.663325    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:10.769419 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:10.663644    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:31:10.884858 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:10.863690    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:10.984958 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:10.982087    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:11.004818 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:11.004392    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:11.140372 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:11.140271    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:31:11.221299 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:11.221254    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:31:11.288389 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:11.288346    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:31:11.382976 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:11.382906    5043 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:31:11.411823 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:11.386820    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx" podUID="f1ca4728-e019-4147-b9a3-d3f662d98e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:11.496990 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:11.496953    5043 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:31:11.497332 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:11.497302    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp" podUID="2299b666-7ae6-452c-bde8-203a6de2e0e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:11.608641 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:11.608548    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:11.664960 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:11.664851    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:11.665679 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:11.665645    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:11.691442 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:11.671335    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:31:11.900770 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:11.871873    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:11.982968 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:11.982921    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:12.021765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:12.009421    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:12.060758 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:12.045266    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77930c58-zfnhc" event={"ID":"4088447a-a476-4088-b3f7-40b825b7ac15","Type":"ContainerStarted","Data":"f8e17778d922277cb06b541402015c88ca9a7e4ff37dbfad9c0230b9897c756c"}
Oct 30 19:31:12.145341 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:12.145299    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:31:12.211644 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:12.211610    5043 generic.go:356] "Generic (PLEG): container finished" podID="4088447a-a476-4088-b3f7-40b825b7ac15" containerID="5484c8aac999fb517b0be2cdf70e7c49447abe856071f8b9eefadda51b3c7719" exitCode=0
Oct 30 19:31:12.215858 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:12.212356    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77930c58-zfnhc" event={"ID":"4088447a-a476-4088-b3f7-40b825b7ac15","Type":"ContainerDied","Data":"5484c8aac999fb517b0be2cdf70e7c49447abe856071f8b9eefadda51b3c7719"}
Oct 30 19:31:12.223141 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:12.223102    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:12.293816 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:12.291830    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:12.630669 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:12.628869    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:12.715761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:12.697012    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:12.753764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:12.717119    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:31:12.827761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:12.762453    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:31:12.923812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:12.919852    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:31:12.923812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:12.922398    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ae6628f-6n765"]
Oct 30 19:31:12.952034 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:12.952008    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ae6628f-6n765"
Oct 30 19:31:12.993634 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:12.986615    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 30 19:31:13.025337 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:13.018471    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ae6628f-6n765"]
Oct 30 19:31:13.025837 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:13.025800    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:13.074090 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:13.074045    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:13.095399 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:13.095351    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbb6d7b2-8922-434c-96f0-650c4c627d63-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ae6628f-6n765\" (UID: \"fbb6d7b2-8922-434c-96f0-650c4c627d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ae6628f-6n765"
Oct 30 19:31:13.110606 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:13.110157    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g85pl\" (UniqueName: \"kubernetes.io/projected/fbb6d7b2-8922-434c-96f0-650c4c627d63-kube-api-access-g85pl\") pod \"maas-api-revoke-keys-e2e-ait-2ae6628f-6n765\" (UID: \"fbb6d7b2-8922-434c-96f0-650c4c627d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ae6628f-6n765"
Oct 30 19:31:13.110606 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:13.110301    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbb6d7b2-8922-434c-96f0-650c4c627d63-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ae6628f-6n765\" (UID: \"fbb6d7b2-8922-434c-96f0-650c4c627d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ae6628f-6n765"
Oct 30 19:31:13.164758 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:13.159409    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:31:13.242817 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:13.221800    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbb6d7b2-8922-434c-96f0-650c4c627d63-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ae6628f-6n765\" (UID: \"fbb6d7b2-8922-434c-96f0-650c4c627d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ae6628f-6n765"
Oct 30 19:31:13.242817 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:13.222050    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g85pl\" (UniqueName: \"kubernetes.io/projected/fbb6d7b2-8922-434c-96f0-650c4c627d63-kube-api-access-g85pl\") pod \"maas-api-revoke-keys-e2e-ait-2ae6628f-6n765\" (UID: \"fbb6d7b2-8922-434c-96f0-650c4c627d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ae6628f-6n765"
Oct 30 19:31:13.242817 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:13.222142    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbb6d7b2-8922-434c-96f0-650c4c627d63-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ae6628f-6n765\" (UID: \"fbb6d7b2-8922-434c-96f0-650c4c627d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ae6628f-6n765"
Oct 30 19:31:13.242817 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:13.225536    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:13.242817 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:13.239590    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbb6d7b2-8922-434c-96f0-650c4c627d63-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ae6628f-6n765\" (UID: \"fbb6d7b2-8922-434c-96f0-650c4c627d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ae6628f-6n765"
Oct 30 19:31:13.274997 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:13.270490    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbb6d7b2-8922-434c-96f0-650c4c627d63-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ae6628f-6n765\" (UID: \"fbb6d7b2-8922-434c-96f0-650c4c627d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ae6628f-6n765"
Oct 30 19:31:13.294801 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:13.293846    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:13.319345 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:13.301415    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g85pl\" (UniqueName: \"kubernetes.io/projected/fbb6d7b2-8922-434c-96f0-650c4c627d63-kube-api-access-g85pl\") pod \"maas-api-revoke-keys-e2e-ait-2ae6628f-6n765\" (UID: \"fbb6d7b2-8922-434c-96f0-650c4c627d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ae6628f-6n765"
Oct 30 19:31:13.534997 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:13.526028    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ae6628f-6n765"
Oct 30 19:31:13.638021 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:13.637901    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:13.640705 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:13.640672    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:13.698923 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:13.698872    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:31:13.842556 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:13.842446    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded"
Oct 30 19:31:13.845539 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:13.842813    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:13.932778 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:13.927158    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:31:13.932778 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:13.929181    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:13.945818 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:13.900983    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:14.059454 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:14.050986    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:31:14.161834 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:14.097182    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:14.161834 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:14.103907    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:31:14.204405 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:14.168865    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:31:14.220961 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:14.220934    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:31:14.290287 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:14.258857    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:31:14.292330 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:14.291826    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"
Oct 30 19:31:14.294637 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:14.294598    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:14.359445 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:14.359421    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"
Oct 30 19:31:14.370685 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:14.367754    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:31:14.370685 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:14.370325    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:31:14.384143 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:14.384111    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:14.388038 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:14.387467    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:31:14.404867 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:14.404821    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:31:14.405674 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:14.405559    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77930c58-zfnhc"
Oct 30 19:31:14.522676 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:14.519826    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4088447a-a476-4088-b3f7-40b825b7ac15-kube-api-access-lz7tg\" (UniqueName: \"kubernetes.io/projected/4088447a-a476-4088-b3f7-40b825b7ac15-kube-api-access-lz7tg\") pod \"4088447a-a476-4088-b3f7-40b825b7ac15\" (UID: \"4088447a-a476-4088-b3f7-40b825b7ac15\") "
Oct 30 19:31:14.522676 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:14.519884    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4088447a-a476-4088-b3f7-40b825b7ac15-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4088447a-a476-4088-b3f7-40b825b7ac15-maas-api-openshift-ca\") pod \"4088447a-a476-4088-b3f7-40b825b7ac15\" (UID: \"4088447a-a476-4088-b3f7-40b825b7ac15\") "
Oct 30 19:31:14.522676 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:14.519918    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4088447a-a476-4088-b3f7-40b825b7ac15-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4088447a-a476-4088-b3f7-40b825b7ac15-maas-api-xks-ca\") pod \"4088447a-a476-4088-b3f7-40b825b7ac15\" (UID: \"4088447a-a476-4088-b3f7-40b825b7ac15\") "
Oct 30 19:31:14.522676 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:14.522535    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4088447a-a476-4088-b3f7-40b825b7ac15-maas-api-openshift-ca" pod "4088447a-a476-4088-b3f7-40b825b7ac15" (UID: "4088447a-a476-4088-b3f7-40b825b7ac15"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:31:14.523704 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:14.523628    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4088447a-a476-4088-b3f7-40b825b7ac15-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:14.525159 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:14.525131    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ae6628f-6n765"]
Oct 30 19:31:14.531756 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:14.531551    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4088447a-a476-4088-b3f7-40b825b7ac15-maas-api-xks-ca" pod "4088447a-a476-4088-b3f7-40b825b7ac15" (UID: "4088447a-a476-4088-b3f7-40b825b7ac15"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:31:14.532801 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:14.532761    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4088447a-a476-4088-b3f7-40b825b7ac15-kube-api-access-lz7tg" pod "4088447a-a476-4088-b3f7-40b825b7ac15" (UID: "4088447a-a476-4088-b3f7-40b825b7ac15"). InnerVolumeSpecName "kube-api-access-lz7tg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:31:14.626045 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:14.626007    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz7tg\" (UniqueName: \"kubernetes.io/projected/4088447a-a476-4088-b3f7-40b825b7ac15-kube-api-access-lz7tg\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:14.626045 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:14.626048    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4088447a-a476-4088-b3f7-40b825b7ac15-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:15.419762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:15.419699    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ae6628f-6n765" event={"ID":"fbb6d7b2-8922-434c-96f0-650c4c627d63","Type":"ContainerStarted","Data":"5463c5cdba3be37cb3047c74cdbc0542165ba1115d9855037f18559ef209a32f"}
Oct 30 19:31:15.420257 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:15.419776    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ae6628f-6n765" event={"ID":"fbb6d7b2-8922-434c-96f0-650c4c627d63","Type":"ContainerStarted","Data":"e2a551f1f581293cbb578f9ed22eeaf48be213075a98e26b3534a56bb941f77a"}
Oct 30 19:31:15.423757 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:15.422778    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77930c58-zfnhc" event={"ID":"4088447a-a476-4088-b3f7-40b825b7ac15","Type":"ContainerDied","Data":"f8e17778d922277cb06b541402015c88ca9a7e4ff37dbfad9c0230b9897c756c"}
Oct 30 19:31:15.423757 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:15.422832    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8e17778d922277cb06b541402015c88ca9a7e4ff37dbfad9c0230b9897c756c"
Oct 30 19:31:15.423757 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:15.422893    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77930c58-zfnhc"
Oct 30 19:31:15.451379 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:15.451326    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ae6628f-6n765" podStartSLOduration=3.451311219 podStartE2EDuration="3.451311219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:15.447405462 +0000 UTC m=+894.958465427" watchObservedRunningTime="2026-10-30 19:31:15.451311219 +0000 UTC m=+894.962371213"
Oct 30 19:31:15.613307 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:15.613279    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-5e333f16-68c977c4bd-rhgmr"
Oct 30 19:31:15.613695 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:15.613669    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5e333f16-f5f6bd6dc-jfs9n"
Oct 30 19:31:15.635750 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:15.635712    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-229db90d-6f45bf4ddb-q875f"]
Oct 30 19:31:15.636442 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:15.636427    5043 cpu_manager.go:414] "containerMap: removing container" podUID="4088447a-a476-4088-b3f7-40b825b7ac15" containerName="revoke-keys"
Oct 30 19:31:15.636490 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:15.636444    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4088447a-a476-4088-b3f7-40b825b7ac15" containerName="revoke-keys"
Oct 30 19:31:15.636579 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:15.636571    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="4088447a-a476-4088-b3f7-40b825b7ac15" containerName="revoke-keys"
Oct 30 19:31:15.640952 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:15.640677    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-229db90d-6f45bf4ddb-q875f"
Oct 30 19:31:15.657319 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:15.657291    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-229db90d-6f45bf4ddb-q875f"]
Oct 30 19:31:15.728701 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:15.728667    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-229db90d\""
Oct 30 19:31:15.729910 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:15.729888    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-229db90d-dockercfg-pcxp2\""
Oct 30 19:31:15.830871 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:15.808450    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f7a8e0a-34a7-4421-adb6-dec8e380d134-plugins-config-volume\") pod \"payload-processing-e2e-ait-229db90d-6f45bf4ddb-q875f\" (UID: \"0f7a8e0a-34a7-4421-adb6-dec8e380d134\") " pod="openshift-ingress/payload-processing-e2e-ait-229db90d-6f45bf4ddb-q875f"
Oct 30 19:31:15.838590 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:15.838555    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzpd5\" (UniqueName: \"kubernetes.io/projected/0f7a8e0a-34a7-4421-adb6-dec8e380d134-kube-api-access-dzpd5\") pod \"payload-processing-e2e-ait-229db90d-6f45bf4ddb-q875f\" (UID: \"0f7a8e0a-34a7-4421-adb6-dec8e380d134\") " pod="openshift-ingress/payload-processing-e2e-ait-229db90d-6f45bf4ddb-q875f"
Oct 30 19:31:15.943821 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:15.943783    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzpd5\" (UniqueName: \"kubernetes.io/projected/0f7a8e0a-34a7-4421-adb6-dec8e380d134-kube-api-access-dzpd5\") pod \"payload-processing-e2e-ait-229db90d-6f45bf4ddb-q875f\" (UID: \"0f7a8e0a-34a7-4421-adb6-dec8e380d134\") " pod="openshift-ingress/payload-processing-e2e-ait-229db90d-6f45bf4ddb-q875f"
Oct 30 19:31:15.944221 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:15.944203    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f7a8e0a-34a7-4421-adb6-dec8e380d134-plugins-config-volume\") pod \"payload-processing-e2e-ait-229db90d-6f45bf4ddb-q875f\" (UID: \"0f7a8e0a-34a7-4421-adb6-dec8e380d134\") " pod="openshift-ingress/payload-processing-e2e-ait-229db90d-6f45bf4ddb-q875f"
Oct 30 19:31:15.945374 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:15.945348    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f7a8e0a-34a7-4421-adb6-dec8e380d134-plugins-config-volume\") pod \"payload-processing-e2e-ait-229db90d-6f45bf4ddb-q875f\" (UID: \"0f7a8e0a-34a7-4421-adb6-dec8e380d134\") " pod="openshift-ingress/payload-processing-e2e-ait-229db90d-6f45bf4ddb-q875f"
Oct 30 19:31:15.974235 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:15.974201    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzpd5\" (UniqueName: \"kubernetes.io/projected/0f7a8e0a-34a7-4421-adb6-dec8e380d134-kube-api-access-dzpd5\") pod \"payload-processing-e2e-ait-229db90d-6f45bf4ddb-q875f\" (UID: \"0f7a8e0a-34a7-4421-adb6-dec8e380d134\") " pod="openshift-ingress/payload-processing-e2e-ait-229db90d-6f45bf4ddb-q875f"
Oct 30 19:31:16.266030 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:16.266002    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-229db90d-6f45bf4ddb-q875f"
Oct 30 19:31:16.434157 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:16.434126    5043 generic.go:356] "Generic (PLEG): container finished" podID="fbb6d7b2-8922-434c-96f0-650c4c627d63" containerID="5463c5cdba3be37cb3047c74cdbc0542165ba1115d9855037f18559ef209a32f" exitCode=0
Oct 30 19:31:16.434543 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:16.434212    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ae6628f-6n765" event={"ID":"fbb6d7b2-8922-434c-96f0-650c4c627d63","Type":"ContainerDied","Data":"5463c5cdba3be37cb3047c74cdbc0542165ba1115d9855037f18559ef209a32f"}
Oct 30 19:31:16.564651 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:16.561907    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-229db90d-6f45bf4ddb-q875f"]
Oct 30 19:31:17.447214 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:17.447175    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-229db90d-6f45bf4ddb-q875f" event={"ID":"0f7a8e0a-34a7-4421-adb6-dec8e380d134","Type":"ContainerStarted","Data":"568d045dc63e0e14488784c2a2e3ee5ffb1c3cfd36cf6983e9791e2b8be94800"}
Oct 30 19:31:17.447679 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:17.447232    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-229db90d-6f45bf4ddb-q875f" event={"ID":"0f7a8e0a-34a7-4421-adb6-dec8e380d134","Type":"ContainerStarted","Data":"2ca0fefa325ec7fb633f267d0e6ddfa7ef9bfc25b1dcb311bb863dbdce09c124"}
Oct 30 19:31:17.447679 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:17.447254    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-229db90d-6f45bf4ddb-q875f"
Oct 30 19:31:17.536866 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:17.526656    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-229db90d-6f45bf4ddb-q875f" podStartSLOduration=2.526636246 podStartE2EDuration="2.526636246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:17.473883146 +0000 UTC m=+896.984943112" watchObservedRunningTime="2026-10-30 19:31:17.526636246 +0000 UTC m=+897.037696210"
Oct 30 19:31:17.774577 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:17.774548    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ae6628f-6n765"
Oct 30 19:31:17.848681 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:17.848650    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbb6d7b2-8922-434c-96f0-650c4c627d63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbb6d7b2-8922-434c-96f0-650c4c627d63-maas-api-openshift-ca\") pod \"fbb6d7b2-8922-434c-96f0-650c4c627d63\" (UID: \"fbb6d7b2-8922-434c-96f0-650c4c627d63\") "
Oct 30 19:31:17.848897 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:17.848746    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbb6d7b2-8922-434c-96f0-650c4c627d63-kube-api-access-g85pl\" (UniqueName: \"kubernetes.io/projected/fbb6d7b2-8922-434c-96f0-650c4c627d63-kube-api-access-g85pl\") pod \"fbb6d7b2-8922-434c-96f0-650c4c627d63\" (UID: \"fbb6d7b2-8922-434c-96f0-650c4c627d63\") "
Oct 30 19:31:17.848977 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:17.848959    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbb6d7b2-8922-434c-96f0-650c4c627d63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbb6d7b2-8922-434c-96f0-650c4c627d63-maas-api-xks-ca\") pod \"fbb6d7b2-8922-434c-96f0-650c4c627d63\" (UID: \"fbb6d7b2-8922-434c-96f0-650c4c627d63\") "
Oct 30 19:31:17.849081 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:17.849052    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbb6d7b2-8922-434c-96f0-650c4c627d63-maas-api-openshift-ca" pod "fbb6d7b2-8922-434c-96f0-650c4c627d63" (UID: "fbb6d7b2-8922-434c-96f0-650c4c627d63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:31:17.849567 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:17.849542    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbb6d7b2-8922-434c-96f0-650c4c627d63-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:17.851204 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:17.851181    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb6d7b2-8922-434c-96f0-650c4c627d63-kube-api-access-g85pl" pod "fbb6d7b2-8922-434c-96f0-650c4c627d63" (UID: "fbb6d7b2-8922-434c-96f0-650c4c627d63"). InnerVolumeSpecName "kube-api-access-g85pl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:31:17.851466 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:17.851449    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbb6d7b2-8922-434c-96f0-650c4c627d63-maas-api-xks-ca" pod "fbb6d7b2-8922-434c-96f0-650c4c627d63" (UID: "fbb6d7b2-8922-434c-96f0-650c4c627d63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:31:17.951017 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:17.950984    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g85pl\" (UniqueName: \"kubernetes.io/projected/fbb6d7b2-8922-434c-96f0-650c4c627d63-kube-api-access-g85pl\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:17.951017 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:17.951013    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbb6d7b2-8922-434c-96f0-650c4c627d63-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:18.458108 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:18.458069    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ae6628f-6n765"
Oct 30 19:31:18.458582 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:18.458101    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ae6628f-6n765" event={"ID":"fbb6d7b2-8922-434c-96f0-650c4c627d63","Type":"ContainerDied","Data":"e2a551f1f581293cbb578f9ed22eeaf48be213075a98e26b3534a56bb941f77a"}
Oct 30 19:31:18.458582 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:18.458189    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2a551f1f581293cbb578f9ed22eeaf48be213075a98e26b3534a56bb941f77a"
Oct 30 19:31:20.856392 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:20.856368    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cmw6w_76e21916-0744-4499-a0cb-64a7a44a8285/console-operator/2.log"
Oct 30 19:31:20.858900 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:20.858881    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cmw6w_76e21916-0744-4499-a0cb-64a7a44a8285/console-operator/2.log"
Oct 30 19:31:22.521582 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:22.521549    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2ae6628f-54ff895c9d-4d9pk"]
Oct 30 19:31:24.521731 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:24.521686    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ae6628f-6n765"]
Oct 30 19:31:24.529646 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:24.529613    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ae6628f-6n765"]
Oct 30 19:31:24.654541 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:24.654499    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbb6d7b2-8922-434c-96f0-650c4c627d63" path="/var/lib/kubelet/pods/fbb6d7b2-8922-434c-96f0-650c4c627d63/volumes"
Oct 30 19:31:24.981887 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:31:24.981845    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:31:27.522798 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:27.522723    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-2ae6628f-54ff895c9d-4d9pk" podUID="ff4784f6-f68b-4183-bd3c-d52592a374c4" containerName="payload-processing" containerID="cri-o://26be335961aebc998e1b13698225cbe47008bbd1a619a856ff25d040ba3abf10" gracePeriod=25
Oct 30 19:31:27.524852 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:27.524823    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7"]
Oct 30 19:31:27.526027 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:27.526003    5043 cpu_manager.go:414] "containerMap: removing container" podUID="fbb6d7b2-8922-434c-96f0-650c4c627d63" containerName="revoke-keys"
Oct 30 19:31:27.526201 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:27.526175    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbb6d7b2-8922-434c-96f0-650c4c627d63" containerName="revoke-keys"
Oct 30 19:31:27.527353 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:27.527328    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbb6d7b2-8922-434c-96f0-650c4c627d63" containerName="revoke-keys"
Oct 30 19:31:27.566633 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:27.566607    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7"
Oct 30 19:31:27.568666 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:27.568635    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7"]
Oct 30 19:31:27.706601 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:27.706566    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ce15b3c-1548-489c-a4f1-2fc151a53d35-workload-certs\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7\" (UID: \"8ce15b3c-1548-489c-a4f1-2fc151a53d35\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7"
Oct 30 19:31:27.706850 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:27.706615    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ce15b3c-1548-489c-a4f1-2fc151a53d35-istio-token\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7\" (UID: \"8ce15b3c-1548-489c-a4f1-2fc151a53d35\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7"
Oct 30 19:31:27.706850 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:27.706805    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ce15b3c-1548-489c-a4f1-2fc151a53d35-istiod-ca-cert\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7\" (UID: \"8ce15b3c-1548-489c-a4f1-2fc151a53d35\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7"
Oct 30 19:31:27.706850 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:27.706834    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ce15b3c-1548-489c-a4f1-2fc151a53d35-istio-podinfo\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7\" (UID: \"8ce15b3c-1548-489c-a4f1-2fc151a53d35\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7"
Oct 30 19:31:27.707041 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:27.706874    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce15b3c-1548-489c-a4f1-2fc151a53d35-credential-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7\" (UID: \"8ce15b3c-1548-489c-a4f1-2fc151a53d35\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7"
Oct 30 19:31:27.707041 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:27.706949    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ce15b3c-1548-489c-a4f1-2fc151a53d35-istio-envoy\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7\" (UID: \"8ce15b3c-1548-489c-a4f1-2fc151a53d35\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7"
Oct 30 19:31:27.707234 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:27.707208    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce15b3c-1548-489c-a4f1-2fc151a53d35-workload-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7\" (UID: \"8ce15b3c-1548-489c-a4f1-2fc151a53d35\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7"
Oct 30 19:31:27.707290 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:27.707258    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ce15b3c-1548-489c-a4f1-2fc151a53d35-istio-data\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7\" (UID: \"8ce15b3c-1548-489c-a4f1-2fc151a53d35\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7"
Oct 30 19:31:27.707393 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:27.707362    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c7sf\" (UniqueName: \"kubernetes.io/projected/8ce15b3c-1548-489c-a4f1-2fc151a53d35-kube-api-access-4c7sf\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7\" (UID: \"8ce15b3c-1548-489c-a4f1-2fc151a53d35\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7"
Oct 30 19:31:27.809129 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:27.809098    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce15b3c-1548-489c-a4f1-2fc151a53d35-workload-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7\" (UID: \"8ce15b3c-1548-489c-a4f1-2fc151a53d35\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7"
Oct 30 19:31:27.809261 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:27.809139    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ce15b3c-1548-489c-a4f1-2fc151a53d35-istio-data\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7\" (UID: \"8ce15b3c-1548-489c-a4f1-2fc151a53d35\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7"
Oct 30 19:31:27.809261 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:27.809199    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c7sf\" (UniqueName: \"kubernetes.io/projected/8ce15b3c-1548-489c-a4f1-2fc151a53d35-kube-api-access-4c7sf\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7\" (UID: \"8ce15b3c-1548-489c-a4f1-2fc151a53d35\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7"
Oct 30 19:31:27.809378 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:27.809307    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ce15b3c-1548-489c-a4f1-2fc151a53d35-workload-certs\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7\" (UID: \"8ce15b3c-1548-489c-a4f1-2fc151a53d35\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7"
Oct 30 19:31:27.809378 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:27.809344    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ce15b3c-1548-489c-a4f1-2fc151a53d35-istio-token\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7\" (UID: \"8ce15b3c-1548-489c-a4f1-2fc151a53d35\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7"
Oct 30 19:31:27.809491 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:27.809450    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ce15b3c-1548-489c-a4f1-2fc151a53d35-istiod-ca-cert\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7\" (UID: \"8ce15b3c-1548-489c-a4f1-2fc151a53d35\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7"
Oct 30 19:31:27.809491 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:27.809484    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ce15b3c-1548-489c-a4f1-2fc151a53d35-istio-podinfo\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7\" (UID: \"8ce15b3c-1548-489c-a4f1-2fc151a53d35\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7"
Oct 30 19:31:27.809599 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:27.809491    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce15b3c-1548-489c-a4f1-2fc151a53d35-workload-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7\" (UID: \"8ce15b3c-1548-489c-a4f1-2fc151a53d35\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7"
Oct 30 19:31:27.809599 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:27.809550    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ce15b3c-1548-489c-a4f1-2fc151a53d35-istio-data\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7\" (UID: \"8ce15b3c-1548-489c-a4f1-2fc151a53d35\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7"
Oct 30 19:31:27.809803 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:27.809783    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ce15b3c-1548-489c-a4f1-2fc151a53d35-workload-certs\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7\" (UID: \"8ce15b3c-1548-489c-a4f1-2fc151a53d35\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7"
Oct 30 19:31:27.809887 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:27.809808    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce15b3c-1548-489c-a4f1-2fc151a53d35-credential-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7\" (UID: \"8ce15b3c-1548-489c-a4f1-2fc151a53d35\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7"
Oct 30 19:31:27.809981 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:27.809961    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ce15b3c-1548-489c-a4f1-2fc151a53d35-istio-envoy\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7\" (UID: \"8ce15b3c-1548-489c-a4f1-2fc151a53d35\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7"
Oct 30 19:31:27.810503 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:27.810415    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ce15b3c-1548-489c-a4f1-2fc151a53d35-istiod-ca-cert\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7\" (UID: \"8ce15b3c-1548-489c-a4f1-2fc151a53d35\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7"
Oct 30 19:31:27.810503 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:27.810463    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce15b3c-1548-489c-a4f1-2fc151a53d35-credential-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7\" (UID: \"8ce15b3c-1548-489c-a4f1-2fc151a53d35\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7"
Oct 30 19:31:27.812608 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:27.812589    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ce15b3c-1548-489c-a4f1-2fc151a53d35-istio-podinfo\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7\" (UID: \"8ce15b3c-1548-489c-a4f1-2fc151a53d35\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7"
Oct 30 19:31:27.812720 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:27.812646    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ce15b3c-1548-489c-a4f1-2fc151a53d35-istio-envoy\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7\" (UID: \"8ce15b3c-1548-489c-a4f1-2fc151a53d35\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7"
Oct 30 19:31:27.817640 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:27.817613    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ce15b3c-1548-489c-a4f1-2fc151a53d35-istio-token\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7\" (UID: \"8ce15b3c-1548-489c-a4f1-2fc151a53d35\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7"
Oct 30 19:31:27.818488 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:27.818467    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c7sf\" (UniqueName: \"kubernetes.io/projected/8ce15b3c-1548-489c-a4f1-2fc151a53d35-kube-api-access-4c7sf\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7\" (UID: \"8ce15b3c-1548-489c-a4f1-2fc151a53d35\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7"
Oct 30 19:31:27.855513 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:27.855039    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2ae6628f-54ff895c9d-4d9pk"
Oct 30 19:31:27.945767 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:27.945210    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7"
Oct 30 19:31:28.014981 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:28.014944    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff4784f6-f68b-4183-bd3c-d52592a374c4-kube-api-access-dvmc9\" (UniqueName: \"kubernetes.io/projected/ff4784f6-f68b-4183-bd3c-d52592a374c4-kube-api-access-dvmc9\") pod \"ff4784f6-f68b-4183-bd3c-d52592a374c4\" (UID: \"ff4784f6-f68b-4183-bd3c-d52592a374c4\") "
Oct 30 19:31:28.015196 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:28.015047    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff4784f6-f68b-4183-bd3c-d52592a374c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff4784f6-f68b-4183-bd3c-d52592a374c4-plugins-config-volume\") pod \"ff4784f6-f68b-4183-bd3c-d52592a374c4\" (UID: \"ff4784f6-f68b-4183-bd3c-d52592a374c4\") "
Oct 30 19:31:28.016909 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:28.016878    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff4784f6-f68b-4183-bd3c-d52592a374c4-plugins-config-volume" pod "ff4784f6-f68b-4183-bd3c-d52592a374c4" (UID: "ff4784f6-f68b-4183-bd3c-d52592a374c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:31:28.021124 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:28.021079    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff4784f6-f68b-4183-bd3c-d52592a374c4-kube-api-access-dvmc9" pod "ff4784f6-f68b-4183-bd3c-d52592a374c4" (UID: "ff4784f6-f68b-4183-bd3c-d52592a374c4"). InnerVolumeSpecName "kube-api-access-dvmc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:31:28.118285 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:28.118250    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvmc9\" (UniqueName: \"kubernetes.io/projected/ff4784f6-f68b-4183-bd3c-d52592a374c4-kube-api-access-dvmc9\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:28.118285 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:28.118289    5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff4784f6-f68b-4183-bd3c-d52592a374c4-plugins-config-volume\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:28.304059 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:31:28.304024    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ce15b3c_1548_489c_a4f1_2fc151a53d35.slice/crio-67365e585d64041578240b803b50091d245fdce4fe0fd824b12fc55db87936b6 WatchSource:0}: Error finding container 67365e585d64041578240b803b50091d245fdce4fe0fd824b12fc55db87936b6: Status 404 returned error can't find the container with id 67365e585d64041578240b803b50091d245fdce4fe0fd824b12fc55db87936b6
Oct 30 19:31:28.306095 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:28.306070    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7"]
Oct 30 19:31:28.306783 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:28.306730    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:31:28.306835 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:28.306818    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:31:28.306871 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:28.306847    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:31:28.461430 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:28.461391    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-229db90d-6f45bf4ddb-q875f"
Oct 30 19:31:28.597696 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:28.597607    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7" event={"ID":"8ce15b3c-1548-489c-a4f1-2fc151a53d35","Type":"ContainerStarted","Data":"5680ee690c20c4258226d6037905090e6336e4a1a7f3ec1f7c9c35d6924fd60c"}
Oct 30 19:31:28.598303 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:28.598272    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7" event={"ID":"8ce15b3c-1548-489c-a4f1-2fc151a53d35","Type":"ContainerStarted","Data":"67365e585d64041578240b803b50091d245fdce4fe0fd824b12fc55db87936b6"}
Oct 30 19:31:28.599823 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:28.599796    5043 generic.go:356] "Generic (PLEG): container finished" podID="ff4784f6-f68b-4183-bd3c-d52592a374c4" containerID="26be335961aebc998e1b13698225cbe47008bbd1a619a856ff25d040ba3abf10" exitCode=0
Oct 30 19:31:28.600017 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:28.600002    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2ae6628f-54ff895c9d-4d9pk" event={"ID":"ff4784f6-f68b-4183-bd3c-d52592a374c4","Type":"ContainerDied","Data":"26be335961aebc998e1b13698225cbe47008bbd1a619a856ff25d040ba3abf10"}
Oct 30 19:31:28.600063 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:28.600050    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2ae6628f-54ff895c9d-4d9pk" event={"ID":"ff4784f6-f68b-4183-bd3c-d52592a374c4","Type":"ContainerDied","Data":"20251be92ee14ff5dbbc149d59c44f860059cd5e858c212e3e1cc7fc683466e3"}
Oct 30 19:31:28.600098 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:28.600081    5043 scope.go:122] "RemoveContainer" containerID="26be335961aebc998e1b13698225cbe47008bbd1a619a856ff25d040ba3abf10"
Oct 30 19:31:28.600241 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:28.600228    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2ae6628f-54ff895c9d-4d9pk"
Oct 30 19:31:28.628061 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:28.628007    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7" podStartSLOduration=1.627989009 podStartE2EDuration="1.627989009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:28.622266133 +0000 UTC m=+908.133326120" watchObservedRunningTime="2026-10-30 19:31:28.627989009 +0000 UTC m=+908.139048975"
Oct 30 19:31:28.631635 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:28.631613    5043 scope.go:122] "RemoveContainer" containerID="26be335961aebc998e1b13698225cbe47008bbd1a619a856ff25d040ba3abf10"
Oct 30 19:31:28.633952 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:31:28.633915    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26be335961aebc998e1b13698225cbe47008bbd1a619a856ff25d040ba3abf10\": container with ID starting with 26be335961aebc998e1b13698225cbe47008bbd1a619a856ff25d040ba3abf10 not found: ID does not exist" containerID="26be335961aebc998e1b13698225cbe47008bbd1a619a856ff25d040ba3abf10"
Oct 30 19:31:28.634075 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:28.633954    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26be335961aebc998e1b13698225cbe47008bbd1a619a856ff25d040ba3abf10"} err="failed to get container status \"26be335961aebc998e1b13698225cbe47008bbd1a619a856ff25d040ba3abf10\": rpc error: code = NotFound desc = could not find container \"26be335961aebc998e1b13698225cbe47008bbd1a619a856ff25d040ba3abf10\": container with ID starting with 26be335961aebc998e1b13698225cbe47008bbd1a619a856ff25d040ba3abf10 not found: ID does not exist"
Oct 30 19:31:28.656699 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:28.656670    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2ae6628f-54ff895c9d-4d9pk"]
Oct 30 19:31:28.656885 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:28.656716    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2ae6628f-54ff895c9d-4d9pk"]
Oct 30 19:31:28.951213 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:28.951127    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7"
Oct 30 19:31:29.240984 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:29.226070    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6"]
Oct 30 19:31:29.240984 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:29.227185    5043 cpu_manager.go:414] "containerMap: removing container" podUID="ff4784f6-f68b-4183-bd3c-d52592a374c4" containerName="payload-processing"
Oct 30 19:31:29.240984 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:29.227211    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff4784f6-f68b-4183-bd3c-d52592a374c4" containerName="payload-processing"
Oct 30 19:31:29.240984 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:29.227485    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff4784f6-f68b-4183-bd3c-d52592a374c4" containerName="payload-processing"
Oct 30 19:31:29.256372 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:29.256336    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6"
Oct 30 19:31:29.262880 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:29.262853    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-395ccc-openshift-default-dockercfg-dj8pr\""
Oct 30 19:31:29.290928 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:29.290893    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6"]
Oct 30 19:31:29.370909 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:29.341831    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-istio-podinfo\") pod \"e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6\" (UID: \"0615f18f-9ed4-4d64-bc4f-1a57e15286cf\") " pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6"
Oct 30 19:31:29.371534 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:29.371484    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-workload-socket\") pod \"e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6\" (UID: \"0615f18f-9ed4-4d64-bc4f-1a57e15286cf\") " pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6"
Oct 30 19:31:29.371910 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:29.371865    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-istio-data\") pod \"e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6\" (UID: \"0615f18f-9ed4-4d64-bc4f-1a57e15286cf\") " pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6"
Oct 30 19:31:29.372308 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:29.372242    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-credential-socket\") pod \"e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6\" (UID: \"0615f18f-9ed4-4d64-bc4f-1a57e15286cf\") " pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6"
Oct 30 19:31:29.372610 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:29.372591    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9mgm\" (UniqueName: \"kubernetes.io/projected/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-kube-api-access-f9mgm\") pod \"e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6\" (UID: \"0615f18f-9ed4-4d64-bc4f-1a57e15286cf\") " pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6"
Oct 30 19:31:29.372877 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:29.372860    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-istiod-ca-cert\") pod \"e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6\" (UID: \"0615f18f-9ed4-4d64-bc4f-1a57e15286cf\") " pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6"
Oct 30 19:31:29.373061 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:29.373036    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-workload-certs\") pod \"e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6\" (UID: \"0615f18f-9ed4-4d64-bc4f-1a57e15286cf\") " pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6"
Oct 30 19:31:29.373236 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:29.373211    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-istio-envoy\") pod \"e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6\" (UID: \"0615f18f-9ed4-4d64-bc4f-1a57e15286cf\") " pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6"
Oct 30 19:31:29.373435 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:29.373421    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-istio-token\") pod \"e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6\" (UID: \"0615f18f-9ed4-4d64-bc4f-1a57e15286cf\") " pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6"
Oct 30 19:31:29.481495 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:29.481459    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-istiod-ca-cert\") pod \"e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6\" (UID: \"0615f18f-9ed4-4d64-bc4f-1a57e15286cf\") " pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6"
Oct 30 19:31:29.481791 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:29.481770    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-workload-certs\") pod \"e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6\" (UID: \"0615f18f-9ed4-4d64-bc4f-1a57e15286cf\") " pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6"
Oct 30 19:31:29.481940 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:29.481925    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-istio-envoy\") pod \"e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6\" (UID: \"0615f18f-9ed4-4d64-bc4f-1a57e15286cf\") " pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6"
Oct 30 19:31:29.482099 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:29.482081    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-istio-token\") pod \"e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6\" (UID: \"0615f18f-9ed4-4d64-bc4f-1a57e15286cf\") " pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6"
Oct 30 19:31:29.482371 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:29.482353    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-istio-podinfo\") pod \"e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6\" (UID: \"0615f18f-9ed4-4d64-bc4f-1a57e15286cf\") " pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6"
Oct 30 19:31:29.482500 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:29.482483    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-workload-socket\") pod \"e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6\" (UID: \"0615f18f-9ed4-4d64-bc4f-1a57e15286cf\") " pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6"
Oct 30 19:31:29.482612 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:29.482600    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-istio-data\") pod \"e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6\" (UID: \"0615f18f-9ed4-4d64-bc4f-1a57e15286cf\") " pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6"
Oct 30 19:31:29.482836 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:29.482811    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-credential-socket\") pod \"e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6\" (UID: \"0615f18f-9ed4-4d64-bc4f-1a57e15286cf\") " pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6"
Oct 30 19:31:29.483041 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:29.483025    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9mgm\" (UniqueName: \"kubernetes.io/projected/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-kube-api-access-f9mgm\") pod \"e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6\" (UID: \"0615f18f-9ed4-4d64-bc4f-1a57e15286cf\") " pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6"
Oct 30 19:31:29.484380 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:29.484335    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-istiod-ca-cert\") pod \"e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6\" (UID: \"0615f18f-9ed4-4d64-bc4f-1a57e15286cf\") " pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6"
Oct 30 19:31:29.485059 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:29.485037    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-workload-socket\") pod \"e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6\" (UID: \"0615f18f-9ed4-4d64-bc4f-1a57e15286cf\") " pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6"
Oct 30 19:31:29.485348 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:29.485321    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-workload-certs\") pod \"e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6\" (UID: \"0615f18f-9ed4-4d64-bc4f-1a57e15286cf\") " pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6"
Oct 30 19:31:29.485573 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:29.485554    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-istio-data\") pod \"e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6\" (UID: \"0615f18f-9ed4-4d64-bc4f-1a57e15286cf\") " pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6"
Oct 30 19:31:29.485973 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:29.485934    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-credential-socket\") pod \"e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6\" (UID: \"0615f18f-9ed4-4d64-bc4f-1a57e15286cf\") " pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6"
Oct 30 19:31:29.496825 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:29.489116    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-istio-podinfo\") pod \"e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6\" (UID: \"0615f18f-9ed4-4d64-bc4f-1a57e15286cf\") " pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6"
Oct 30 19:31:29.528697 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:29.524703    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-istio-envoy\") pod \"e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6\" (UID: \"0615f18f-9ed4-4d64-bc4f-1a57e15286cf\") " pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6"
Oct 30 19:31:29.528697 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:29.525373    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9mgm\" (UniqueName: \"kubernetes.io/projected/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-kube-api-access-f9mgm\") pod \"e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6\" (UID: \"0615f18f-9ed4-4d64-bc4f-1a57e15286cf\") " pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6"
Oct 30 19:31:29.535136 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:29.535104    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-istio-token\") pod \"e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6\" (UID: \"0615f18f-9ed4-4d64-bc4f-1a57e15286cf\") " pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6"
Oct 30 19:31:29.644036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:29.644003    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6"
Oct 30 19:31:29.947385 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:29.947348    5043 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 19:31:29.947590 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:29.947413    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7" podUID="8ce15b3c-1548-489c-a4f1-2fc151a53d35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 19:31:30.049532 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:31:30.049489    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0615f18f_9ed4_4d64_bc4f_1a57e15286cf.slice/crio-a51a3728d77832f32bf43e4944c4a303eaf2c186a8e9d13549d8b99153d0e5f1 WatchSource:0}: Error finding container a51a3728d77832f32bf43e4944c4a303eaf2c186a8e9d13549d8b99153d0e5f1: Status 404 returned error can't find the container with id a51a3728d77832f32bf43e4944c4a303eaf2c186a8e9d13549d8b99153d0e5f1
Oct 30 19:31:30.051271 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:30.051214    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6"]
Oct 30 19:31:30.052435 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:30.052398    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:31:30.052523 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:30.052493    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:31:30.052565 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:30.052537    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:31:30.682763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:30.681270    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff4784f6-f68b-4183-bd3c-d52592a374c4" path="/var/lib/kubelet/pods/ff4784f6-f68b-4183-bd3c-d52592a374c4/volumes"
Oct 30 19:31:30.682763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:30.681821    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6" event={"ID":"0615f18f-9ed4-4d64-bc4f-1a57e15286cf","Type":"ContainerStarted","Data":"c583cc7aee0b638dc10d53a5b958f084610eede17043d5993b893fe520295561"}
Oct 30 19:31:30.682763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:30.681878    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6" event={"ID":"0615f18f-9ed4-4d64-bc4f-1a57e15286cf","Type":"ContainerStarted","Data":"a51a3728d77832f32bf43e4944c4a303eaf2c186a8e9d13549d8b99153d0e5f1"}
Oct 30 19:31:30.707543 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:30.707487    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6" podStartSLOduration=2.707472953 podStartE2EDuration="2.707472953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:30.700056284 +0000 UTC m=+910.211116250" watchObservedRunningTime="2026-10-30 19:31:30.707472953 +0000 UTC m=+910.218532918"
Oct 30 19:31:30.946191 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:30.946093    5043 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 19:31:30.946191 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:30.946171    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7" podUID="8ce15b3c-1548-489c-a4f1-2fc151a53d35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 19:31:31.653798 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:31.649092    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6"
Oct 30 19:31:31.720765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:31.718010    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6"
Oct 30 19:31:31.744765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:31.736499    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6"
Oct 30 19:31:31.744765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:31.737995    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6"
Oct 30 19:31:31.968769 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:31.963532    5043 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 19:31:31.968769 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:31.963600    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7" podUID="8ce15b3c-1548-489c-a4f1-2fc151a53d35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 19:31:32.025135 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:31.978932    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77930c58-zfnhc"]
Oct 30 19:31:32.025135 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:31.996961    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77930c58-zfnhc"]
Oct 30 19:31:32.677761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:32.657886    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4088447a-a476-4088-b3f7-40b825b7ac15" path="/var/lib/kubelet/pods/4088447a-a476-4088-b3f7-40b825b7ac15/volumes"
Oct 30 19:31:32.949809 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:32.946978    5043 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 19:31:32.949809 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:32.947037    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7" podUID="8ce15b3c-1548-489c-a4f1-2fc151a53d35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 19:31:33.946266 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:33.946223    5043 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 19:31:33.946457 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:33.946281    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7" podUID="8ce15b3c-1548-489c-a4f1-2fc151a53d35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 19:31:34.946928 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:34.946891    5043 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 19:31:34.947341 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:34.946961    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7" podUID="8ce15b3c-1548-489c-a4f1-2fc151a53d35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 19:31:35.290750 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:35.290690    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-395ccc-7d4b79d8f9-gn47s"]
Oct 30 19:31:35.308847 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:35.306961    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-395ccc-7d4b79d8f9-gn47s"
Oct 30 19:31:35.331790 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:35.331694    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b3cc611-c18a-49d7-a09c-5185d7ee2ea4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-395ccc-7d4b79d8f9-gn47s\" (UID: \"5b3cc611-c18a-49d7-a09c-5185d7ee2ea4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-395ccc-7d4b79d8f9-gn47s"
Oct 30 19:31:35.337845 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:35.337795    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-395ccc-7d4b79d8f9-gn47s"]
Oct 30 19:31:35.365046 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:35.365008    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vqhw\" (UniqueName: \"kubernetes.io/projected/5b3cc611-c18a-49d7-a09c-5185d7ee2ea4-kube-api-access-2vqhw\") pod \"payload-pre-processing-e2e-shared-a-w0-395ccc-7d4b79d8f9-gn47s\" (UID: \"5b3cc611-c18a-49d7-a09c-5185d7ee2ea4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-395ccc-7d4b79d8f9-gn47s"
Oct 30 19:31:35.436842 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:35.417430    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-395ccc\""
Oct 30 19:31:35.452877 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:35.420968    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-395ccc-dockercfg-v4s55\""
Oct 30 19:31:35.510865 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:35.504008    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vqhw\" (UniqueName: \"kubernetes.io/projected/5b3cc611-c18a-49d7-a09c-5185d7ee2ea4-kube-api-access-2vqhw\") pod \"payload-pre-processing-e2e-shared-a-w0-395ccc-7d4b79d8f9-gn47s\" (UID: \"5b3cc611-c18a-49d7-a09c-5185d7ee2ea4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-395ccc-7d4b79d8f9-gn47s"
Oct 30 19:31:35.511579 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:35.511553    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b3cc611-c18a-49d7-a09c-5185d7ee2ea4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-395ccc-7d4b79d8f9-gn47s\" (UID: \"5b3cc611-c18a-49d7-a09c-5185d7ee2ea4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-395ccc-7d4b79d8f9-gn47s"
Oct 30 19:31:35.513149 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:35.513123    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b3cc611-c18a-49d7-a09c-5185d7ee2ea4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-395ccc-7d4b79d8f9-gn47s\" (UID: \"5b3cc611-c18a-49d7-a09c-5185d7ee2ea4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-395ccc-7d4b79d8f9-gn47s"
Oct 30 19:31:35.551994 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:35.551707    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vqhw\" (UniqueName: \"kubernetes.io/projected/5b3cc611-c18a-49d7-a09c-5185d7ee2ea4-kube-api-access-2vqhw\") pod \"payload-pre-processing-e2e-shared-a-w0-395ccc-7d4b79d8f9-gn47s\" (UID: \"5b3cc611-c18a-49d7-a09c-5185d7ee2ea4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-395ccc-7d4b79d8f9-gn47s"
Oct 30 19:31:35.581420 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:35.581375    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-395ccc-7d4b79d8f9-gn47s"
Oct 30 19:31:35.949861 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:35.949762    5043 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 30 19:31:35.949861 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:35.949846    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7" podUID="8ce15b3c-1548-489c-a4f1-2fc151a53d35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 30 19:31:36.005126 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:36.005092    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-395ccc-7d4b79d8f9-gn47s"]
Oct 30 19:31:36.837776 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:36.837719    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-395ccc-7d4b79d8f9-gn47s" event={"ID":"5b3cc611-c18a-49d7-a09c-5185d7ee2ea4","Type":"ContainerStarted","Data":"7d086db4b0c80017a7f15a80e4f4916a65195bd169eba686dd5ddb2716f5a698"}
Oct 30 19:31:36.837997 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:36.837801    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-395ccc-7d4b79d8f9-gn47s" event={"ID":"5b3cc611-c18a-49d7-a09c-5185d7ee2ea4","Type":"ContainerStarted","Data":"4e6da541b4566f74fd18a6c47600228ceda9a29d296d8580fee52b9891b9bf36"}
Oct 30 19:31:36.837997 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:36.837826    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-395ccc-7d4b79d8f9-gn47s"
Oct 30 19:31:36.888379 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:36.888302    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-395ccc-7d4b79d8f9-gn47s" podStartSLOduration=1.88828169 podStartE2EDuration="1.88828169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:36.885083861 +0000 UTC m=+916.396143826" watchObservedRunningTime="2026-10-30 19:31:36.88828169 +0000 UTC m=+916.399341656"
Oct 30 19:31:37.129233 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:37.129131    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7"
Oct 30 19:31:37.129752 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:37.129689    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7"
Oct 30 19:31:37.130982 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:37.130950    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7"
Oct 30 19:31:37.306081 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:37.306051    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5e333f16-5j9n4"]
Oct 30 19:31:37.312600 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:37.312560    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5e333f16-5j9n4"
Oct 30 19:31:37.317659 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:37.317627    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:31:37.317867 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:37.317758    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:31:37.317867 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:37.317779    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:31:37.317867 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:37.317637    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5e333f16-5j9n4"]
Oct 30 19:31:37.317867 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:37.317836    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:31:37.358681 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:37.358644    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gmvt\" (UniqueName: \"kubernetes.io/projected/c06745d3-110e-4228-ac45-4b806095300c-kube-api-access-7gmvt\") pod \"maas-api-revoke-keys-e2e-migrate-5e333f16-5j9n4\" (UID: \"c06745d3-110e-4228-ac45-4b806095300c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5e333f16-5j9n4"
Oct 30 19:31:37.359070 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:37.359044    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c06745d3-110e-4228-ac45-4b806095300c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5e333f16-5j9n4\" (UID: \"c06745d3-110e-4228-ac45-4b806095300c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5e333f16-5j9n4"
Oct 30 19:31:37.359197 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:37.359117    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c06745d3-110e-4228-ac45-4b806095300c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5e333f16-5j9n4\" (UID: \"c06745d3-110e-4228-ac45-4b806095300c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5e333f16-5j9n4"
Oct 30 19:31:37.461506 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:37.461467    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gmvt\" (UniqueName: \"kubernetes.io/projected/c06745d3-110e-4228-ac45-4b806095300c-kube-api-access-7gmvt\") pod \"maas-api-revoke-keys-e2e-migrate-5e333f16-5j9n4\" (UID: \"c06745d3-110e-4228-ac45-4b806095300c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5e333f16-5j9n4"
Oct 30 19:31:37.461768 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:37.461748    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c06745d3-110e-4228-ac45-4b806095300c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5e333f16-5j9n4\" (UID: \"c06745d3-110e-4228-ac45-4b806095300c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5e333f16-5j9n4"
Oct 30 19:31:37.462007 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:37.461964    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c06745d3-110e-4228-ac45-4b806095300c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5e333f16-5j9n4\" (UID: \"c06745d3-110e-4228-ac45-4b806095300c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5e333f16-5j9n4"
Oct 30 19:31:37.462617 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:37.462590    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c06745d3-110e-4228-ac45-4b806095300c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5e333f16-5j9n4\" (UID: \"c06745d3-110e-4228-ac45-4b806095300c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5e333f16-5j9n4"
Oct 30 19:31:37.464778 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:37.464727    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c06745d3-110e-4228-ac45-4b806095300c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5e333f16-5j9n4\" (UID: \"c06745d3-110e-4228-ac45-4b806095300c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5e333f16-5j9n4"
Oct 30 19:31:37.476062 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:37.476037    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gmvt\" (UniqueName: \"kubernetes.io/projected/c06745d3-110e-4228-ac45-4b806095300c-kube-api-access-7gmvt\") pod \"maas-api-revoke-keys-e2e-migrate-5e333f16-5j9n4\" (UID: \"c06745d3-110e-4228-ac45-4b806095300c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5e333f16-5j9n4"
Oct 30 19:31:37.637601 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:37.637569    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5e333f16-5j9n4"
Oct 30 19:31:37.938152 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:31:37.938122    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc06745d3_110e_4228_ac45_4b806095300c.slice/crio-ea1bc46771c3523b6a8ecc0e05b92211619bfecf679579a9a5c6fda50d8ae80b WatchSource:0}: Error finding container ea1bc46771c3523b6a8ecc0e05b92211619bfecf679579a9a5c6fda50d8ae80b: Status 404 returned error can't find the container with id ea1bc46771c3523b6a8ecc0e05b92211619bfecf679579a9a5c6fda50d8ae80b
Oct 30 19:31:37.942830 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:37.942806    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5e333f16-5j9n4"]
Oct 30 19:31:38.871793 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:38.866872    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5e333f16-5j9n4" event={"ID":"c06745d3-110e-4228-ac45-4b806095300c","Type":"ContainerStarted","Data":"e3a2ede7c20726db7d06d5bbfa60b1d825d1ff886ed5a5817b64fbeeac23bc75"}
Oct 30 19:31:38.871793 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:38.866965    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5e333f16-5j9n4" event={"ID":"c06745d3-110e-4228-ac45-4b806095300c","Type":"ContainerStarted","Data":"ea1bc46771c3523b6a8ecc0e05b92211619bfecf679579a9a5c6fda50d8ae80b"}
Oct 30 19:31:38.891849 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:38.891787    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5e333f16-5j9n4" podStartSLOduration=1.891767519 podStartE2EDuration="1.891767519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:38.887376325 +0000 UTC m=+918.398436291" watchObservedRunningTime="2026-10-30 19:31:38.891767519 +0000 UTC m=+918.402827486"
Oct 30 19:31:39.900053 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:39.899876    5043 generic.go:356] "Generic (PLEG): container finished" podID="c06745d3-110e-4228-ac45-4b806095300c" containerID="e3a2ede7c20726db7d06d5bbfa60b1d825d1ff886ed5a5817b64fbeeac23bc75" exitCode=0
Oct 30 19:31:39.900053 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:39.899988    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5e333f16-5j9n4" event={"ID":"c06745d3-110e-4228-ac45-4b806095300c","Type":"ContainerDied","Data":"e3a2ede7c20726db7d06d5bbfa60b1d825d1ff886ed5a5817b64fbeeac23bc75"}
Oct 30 19:31:41.111053 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:41.111028    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5e333f16-5j9n4"
Oct 30 19:31:41.169121 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:41.169029    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c06745d3-110e-4228-ac45-4b806095300c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c06745d3-110e-4228-ac45-4b806095300c-maas-api-xks-ca\") pod \"c06745d3-110e-4228-ac45-4b806095300c\" (UID: \"c06745d3-110e-4228-ac45-4b806095300c\") "
Oct 30 19:31:41.184387 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:41.179592    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c06745d3-110e-4228-ac45-4b806095300c-kube-api-access-7gmvt\" (UniqueName: \"kubernetes.io/projected/c06745d3-110e-4228-ac45-4b806095300c-kube-api-access-7gmvt\") pod \"c06745d3-110e-4228-ac45-4b806095300c\" (UID: \"c06745d3-110e-4228-ac45-4b806095300c\") "
Oct 30 19:31:41.184387 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:41.179825    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c06745d3-110e-4228-ac45-4b806095300c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c06745d3-110e-4228-ac45-4b806095300c-maas-api-openshift-ca\") pod \"c06745d3-110e-4228-ac45-4b806095300c\" (UID: \"c06745d3-110e-4228-ac45-4b806095300c\") "
Oct 30 19:31:41.184387 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:41.181212    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c06745d3-110e-4228-ac45-4b806095300c-maas-api-openshift-ca" pod "c06745d3-110e-4228-ac45-4b806095300c" (UID: "c06745d3-110e-4228-ac45-4b806095300c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:31:41.195154 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:41.186706    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c06745d3-110e-4228-ac45-4b806095300c-kube-api-access-7gmvt" pod "c06745d3-110e-4228-ac45-4b806095300c" (UID: "c06745d3-110e-4228-ac45-4b806095300c"). InnerVolumeSpecName "kube-api-access-7gmvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:31:41.198005 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:41.197232    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c06745d3-110e-4228-ac45-4b806095300c-maas-api-xks-ca" pod "c06745d3-110e-4228-ac45-4b806095300c" (UID: "c06745d3-110e-4228-ac45-4b806095300c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:31:41.282451 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:41.282415    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c06745d3-110e-4228-ac45-4b806095300c-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:41.282451 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:41.282449    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gmvt\" (UniqueName: \"kubernetes.io/projected/c06745d3-110e-4228-ac45-4b806095300c-kube-api-access-7gmvt\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:41.282648 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:41.282461    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c06745d3-110e-4228-ac45-4b806095300c-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:41.931202 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:41.931158    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5e333f16-5j9n4" event={"ID":"c06745d3-110e-4228-ac45-4b806095300c","Type":"ContainerDied","Data":"ea1bc46771c3523b6a8ecc0e05b92211619bfecf679579a9a5c6fda50d8ae80b"}
Oct 30 19:31:41.931459 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:41.931269    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea1bc46771c3523b6a8ecc0e05b92211619bfecf679579a9a5c6fda50d8ae80b"
Oct 30 19:31:41.931459 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:41.931354    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5e333f16-5j9n4"
Oct 30 19:31:44.345816 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:44.345769    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-229db90d-bds9s"]
Oct 30 19:31:44.346504 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:44.346488    5043 cpu_manager.go:414] "containerMap: removing container" podUID="c06745d3-110e-4228-ac45-4b806095300c" containerName="revoke-keys"
Oct 30 19:31:44.346550 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:44.346505    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c06745d3-110e-4228-ac45-4b806095300c" containerName="revoke-keys"
Oct 30 19:31:44.346660 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:44.346651    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="c06745d3-110e-4228-ac45-4b806095300c" containerName="revoke-keys"
Oct 30 19:31:44.350993 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:44.350972    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-229db90d-bds9s"
Oct 30 19:31:44.355118 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:44.355096    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:31:44.356123 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:44.356099    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:31:44.356257 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:44.356104    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:31:44.356257 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:44.356188    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:31:44.360970 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:44.360945    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-229db90d-bds9s"]
Oct 30 19:31:44.433820 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:44.433729    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4kw9\" (UniqueName: \"kubernetes.io/projected/dfc1cab2-465a-4792-a639-2a5a2ddbf6ca-kube-api-access-r4kw9\") pod \"maas-api-revoke-keys-e2e-ait-229db90d-bds9s\" (UID: \"dfc1cab2-465a-4792-a639-2a5a2ddbf6ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-229db90d-bds9s"
Oct 30 19:31:44.434031 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:44.433949    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc1cab2-465a-4792-a639-2a5a2ddbf6ca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-229db90d-bds9s\" (UID: \"dfc1cab2-465a-4792-a639-2a5a2ddbf6ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-229db90d-bds9s"
Oct 30 19:31:44.434241 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:44.434220    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc1cab2-465a-4792-a639-2a5a2ddbf6ca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-229db90d-bds9s\" (UID: \"dfc1cab2-465a-4792-a639-2a5a2ddbf6ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-229db90d-bds9s"
Oct 30 19:31:44.541160 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:44.541117    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc1cab2-465a-4792-a639-2a5a2ddbf6ca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-229db90d-bds9s\" (UID: \"dfc1cab2-465a-4792-a639-2a5a2ddbf6ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-229db90d-bds9s"
Oct 30 19:31:44.557680 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:44.557641    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4kw9\" (UniqueName: \"kubernetes.io/projected/dfc1cab2-465a-4792-a639-2a5a2ddbf6ca-kube-api-access-r4kw9\") pod \"maas-api-revoke-keys-e2e-ait-229db90d-bds9s\" (UID: \"dfc1cab2-465a-4792-a639-2a5a2ddbf6ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-229db90d-bds9s"
Oct 30 19:31:44.569505 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:44.558659    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc1cab2-465a-4792-a639-2a5a2ddbf6ca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-229db90d-bds9s\" (UID: \"dfc1cab2-465a-4792-a639-2a5a2ddbf6ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-229db90d-bds9s"
Oct 30 19:31:44.574475 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:44.569576    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc1cab2-465a-4792-a639-2a5a2ddbf6ca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-229db90d-bds9s\" (UID: \"dfc1cab2-465a-4792-a639-2a5a2ddbf6ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-229db90d-bds9s"
Oct 30 19:31:44.579307 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:44.570207    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc1cab2-465a-4792-a639-2a5a2ddbf6ca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-229db90d-bds9s\" (UID: \"dfc1cab2-465a-4792-a639-2a5a2ddbf6ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-229db90d-bds9s"
Oct 30 19:31:44.608826 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:44.601439    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4kw9\" (UniqueName: \"kubernetes.io/projected/dfc1cab2-465a-4792-a639-2a5a2ddbf6ca-kube-api-access-r4kw9\") pod \"maas-api-revoke-keys-e2e-ait-229db90d-bds9s\" (UID: \"dfc1cab2-465a-4792-a639-2a5a2ddbf6ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-229db90d-bds9s"
Oct 30 19:31:44.681588 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:44.681318    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-229db90d-bds9s"
Oct 30 19:31:45.356985 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:31:45.356953    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfc1cab2_465a_4792_a639_2a5a2ddbf6ca.slice/crio-4f0100952088a0781fe74508aceadbfb8120dc8d5bd2c37e8ba3bde8141a300b WatchSource:0}: Error finding container 4f0100952088a0781fe74508aceadbfb8120dc8d5bd2c37e8ba3bde8141a300b: Status 404 returned error can't find the container with id 4f0100952088a0781fe74508aceadbfb8120dc8d5bd2c37e8ba3bde8141a300b
Oct 30 19:31:45.361812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:45.361788    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-229db90d-bds9s"]
Oct 30 19:31:46.159689 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:46.158344    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-229db90d-bds9s" event={"ID":"dfc1cab2-465a-4792-a639-2a5a2ddbf6ca","Type":"ContainerStarted","Data":"0776b546f768c72deec0cf46a07cc28d49ee2bc972d8fde2f7c62913aed24d25"}
Oct 30 19:31:46.159689 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:46.158453    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-229db90d-bds9s" event={"ID":"dfc1cab2-465a-4792-a639-2a5a2ddbf6ca","Type":"ContainerStarted","Data":"4f0100952088a0781fe74508aceadbfb8120dc8d5bd2c37e8ba3bde8141a300b"}
Oct 30 19:31:46.184164 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:46.184117    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-229db90d-bds9s" podStartSLOduration=2.18409893 podStartE2EDuration="2.18409893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:46.181185916 +0000 UTC m=+925.692245882" watchObservedRunningTime="2026-10-30 19:31:46.18409893 +0000 UTC m=+925.695158900"
Oct 30 19:31:47.209312 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:47.209272    5043 generic.go:356] "Generic (PLEG): container finished" podID="dfc1cab2-465a-4792-a639-2a5a2ddbf6ca" containerID="0776b546f768c72deec0cf46a07cc28d49ee2bc972d8fde2f7c62913aed24d25" exitCode=0
Oct 30 19:31:47.209975 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:47.209949    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-229db90d-bds9s" event={"ID":"dfc1cab2-465a-4792-a639-2a5a2ddbf6ca","Type":"ContainerDied","Data":"0776b546f768c72deec0cf46a07cc28d49ee2bc972d8fde2f7c62913aed24d25"}
Oct 30 19:31:47.595824 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:47.591489    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:47.669763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:47.669166    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:47.718765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:47.717895    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:47.749128 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:47.743952    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:47.876070 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:47.875968    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-395ccc-7d4b79d8f9-gn47s"
Oct 30 19:31:48.043915 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:48.043864    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:48.048762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:48.044918    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:48.663990 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:48.602854    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:48.663990 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:48.635830    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:48.663990 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:48.636344    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"
Oct 30 19:31:48.745939 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:48.745888    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:48.820988 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:48.804870    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:48.870385 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:48.804917    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:31:48.934859 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:48.805905    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5e333f16-68c977c4bd-rhgmr"]
Oct 30 19:31:48.934859 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:48.884857    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:31:48.934859 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:48.885071    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:31:49.183789 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:49.166066    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded"
Oct 30 19:31:49.183789 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:49.169838    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:31:49.243761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:49.226693    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:31:49.342774 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:49.264855    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:49.342774 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:49.290088    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:49.665029 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:49.664908    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:49.873887 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:49.853975    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:31:50.033761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:50.030623    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:50.033761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:50.031974    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:50.227304 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:50.224393    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": context deadline exceeded"
Oct 30 19:31:50.227304 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:50.224667    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:50.317821 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:50.311277    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:31:50.430860 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:50.430823    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:50.576368 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:50.571674    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:31:50.734044 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:31:50.734000    5043 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.849s"
Oct 30 19:31:50.743895 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:50.743853    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:31:50.768764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:50.752314    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:31:50.768764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:50.754633    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:31:50.768764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:50.754661    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5e333f16-f5f6bd6dc-jfs9n"]
Oct 30 19:31:50.768764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:50.754810    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:31:50.768764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:50.754851    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"
Oct 30 19:31:50.768764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:50.754908    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:31:50.903824 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:50.902919    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-229db90d-bds9s"
Oct 30 19:31:50.928643 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:50.928596    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:50.929322 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:50.928685    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded"
Oct 30 19:31:51.045252 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:51.045222    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfc1cab2-465a-4792-a639-2a5a2ddbf6ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc1cab2-465a-4792-a639-2a5a2ddbf6ca-maas-api-openshift-ca\") pod \"dfc1cab2-465a-4792-a639-2a5a2ddbf6ca\" (UID: \"dfc1cab2-465a-4792-a639-2a5a2ddbf6ca\") "
Oct 30 19:31:51.046088 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:51.046064    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfc1cab2-465a-4792-a639-2a5a2ddbf6ca-kube-api-access-r4kw9\" (UniqueName: \"kubernetes.io/projected/dfc1cab2-465a-4792-a639-2a5a2ddbf6ca-kube-api-access-r4kw9\") pod \"dfc1cab2-465a-4792-a639-2a5a2ddbf6ca\" (UID: \"dfc1cab2-465a-4792-a639-2a5a2ddbf6ca\") "
Oct 30 19:31:51.046546 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:51.046501    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfc1cab2-465a-4792-a639-2a5a2ddbf6ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc1cab2-465a-4792-a639-2a5a2ddbf6ca-maas-api-xks-ca\") pod \"dfc1cab2-465a-4792-a639-2a5a2ddbf6ca\" (UID: \"dfc1cab2-465a-4792-a639-2a5a2ddbf6ca\") "
Oct 30 19:31:51.051340 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:51.051311    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:31:51.051484 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:51.051393    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:31:51.051484 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:51.051449    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:31:51.051484 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:51.051466    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"
Oct 30 19:31:51.051660 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:51.051516    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:31:51.051660 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:51.051565    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:31:51.051660 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:51.051607    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:31:51.051660 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:51.051649    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:31:51.051884 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:51.051694    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:31:51.051884 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:51.051766    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:31:51.055718 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:51.055693    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfc1cab2-465a-4792-a639-2a5a2ddbf6ca-maas-api-openshift-ca" pod "dfc1cab2-465a-4792-a639-2a5a2ddbf6ca" (UID: "dfc1cab2-465a-4792-a639-2a5a2ddbf6ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:31:51.140650 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:51.114052    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfc1cab2-465a-4792-a639-2a5a2ddbf6ca-maas-api-xks-ca" pod "dfc1cab2-465a-4792-a639-2a5a2ddbf6ca" (UID: "dfc1cab2-465a-4792-a639-2a5a2ddbf6ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:31:51.141230 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:51.141182    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfc1cab2-465a-4792-a639-2a5a2ddbf6ca-kube-api-access-r4kw9" pod "dfc1cab2-465a-4792-a639-2a5a2ddbf6ca" (UID: "dfc1cab2-465a-4792-a639-2a5a2ddbf6ca"). InnerVolumeSpecName "kube-api-access-r4kw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:31:51.222970 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:51.199594    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:31:51.252526 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:51.252490    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc1cab2-465a-4792-a639-2a5a2ddbf6ca-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:51.252785 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:51.252768    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4kw9\" (UniqueName: \"kubernetes.io/projected/dfc1cab2-465a-4792-a639-2a5a2ddbf6ca-kube-api-access-r4kw9\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:51.252889 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:51.252878    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc1cab2-465a-4792-a639-2a5a2ddbf6ca-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:51.253502 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:51.253485    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"
Oct 30 19:31:51.618923 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:51.613243    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-229db90d-bds9s"
Oct 30 19:31:51.619155 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:51.614601    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-229db90d-bds9s" event={"ID":"dfc1cab2-465a-4792-a639-2a5a2ddbf6ca","Type":"ContainerDied","Data":"4f0100952088a0781fe74508aceadbfb8120dc8d5bd2c37e8ba3bde8141a300b"}
Oct 30 19:31:51.623416 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:51.623387    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f0100952088a0781fe74508aceadbfb8120dc8d5bd2c37e8ba3bde8141a300b"
Oct 30 19:31:53.967180 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:53.967132    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-5e333f16-68c977c4bd-rhgmr" podUID="0669101d-0bf5-4c6c-a088-180749a12376" containerName="payload-processing" containerID="cri-o://723f67b7845b5f44754fa62fa12c0f32f810de33f8d63db56f316d39d4329578" gracePeriod=25
Oct 30 19:31:54.462912 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:54.462390    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-5e333f16-68c977c4bd-rhgmr"
Oct 30 19:31:54.509663 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:54.509607    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-229db90d-6f45bf4ddb-q875f"]
Oct 30 19:31:54.613671 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:54.610578    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0669101d-0bf5-4c6c-a088-180749a12376-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0669101d-0bf5-4c6c-a088-180749a12376-plugins-config-volume\") pod \"0669101d-0bf5-4c6c-a088-180749a12376\" (UID: \"0669101d-0bf5-4c6c-a088-180749a12376\") "
Oct 30 19:31:54.613671 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:54.612472    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0669101d-0bf5-4c6c-a088-180749a12376-kube-api-access-vspmb\" (UniqueName: \"kubernetes.io/projected/0669101d-0bf5-4c6c-a088-180749a12376-kube-api-access-vspmb\") pod \"0669101d-0bf5-4c6c-a088-180749a12376\" (UID: \"0669101d-0bf5-4c6c-a088-180749a12376\") "
Oct 30 19:31:54.613671 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:54.613195    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0669101d-0bf5-4c6c-a088-180749a12376-plugins-config-volume" pod "0669101d-0bf5-4c6c-a088-180749a12376" (UID: "0669101d-0bf5-4c6c-a088-180749a12376"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:31:54.613671 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:54.613380    5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0669101d-0bf5-4c6c-a088-180749a12376-plugins-config-volume\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:54.625491 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:54.621653    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0669101d-0bf5-4c6c-a088-180749a12376-kube-api-access-vspmb" pod "0669101d-0bf5-4c6c-a088-180749a12376" (UID: "0669101d-0bf5-4c6c-a088-180749a12376"). InnerVolumeSpecName "kube-api-access-vspmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:31:54.714639 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:54.714600    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vspmb\" (UniqueName: \"kubernetes.io/projected/0669101d-0bf5-4c6c-a088-180749a12376-kube-api-access-vspmb\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:54.733424 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:54.733384    5043 generic.go:356] "Generic (PLEG): container finished" podID="0669101d-0bf5-4c6c-a088-180749a12376" containerID="723f67b7845b5f44754fa62fa12c0f32f810de33f8d63db56f316d39d4329578" exitCode=0
Oct 30 19:31:54.733598 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:54.733442    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-5e333f16-68c977c4bd-rhgmr"
Oct 30 19:31:54.733598 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:54.733476    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-5e333f16-68c977c4bd-rhgmr" event={"ID":"0669101d-0bf5-4c6c-a088-180749a12376","Type":"ContainerDied","Data":"723f67b7845b5f44754fa62fa12c0f32f810de33f8d63db56f316d39d4329578"}
Oct 30 19:31:54.733598 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:54.733551    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-5e333f16-68c977c4bd-rhgmr" event={"ID":"0669101d-0bf5-4c6c-a088-180749a12376","Type":"ContainerDied","Data":"810bf71e3736846a362fd0c4dcc47f27eb92bcee8c89c67fbe3239980d6da830"}
Oct 30 19:31:54.733598 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:54.733568    5043 scope.go:122] "RemoveContainer" containerID="723f67b7845b5f44754fa62fa12c0f32f810de33f8d63db56f316d39d4329578"
Oct 30 19:31:54.749432 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:54.749400    5043 scope.go:122] "RemoveContainer" containerID="723f67b7845b5f44754fa62fa12c0f32f810de33f8d63db56f316d39d4329578"
Oct 30 19:31:54.749921 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:31:54.749883    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"723f67b7845b5f44754fa62fa12c0f32f810de33f8d63db56f316d39d4329578\": container with ID starting with 723f67b7845b5f44754fa62fa12c0f32f810de33f8d63db56f316d39d4329578 not found: ID does not exist" containerID="723f67b7845b5f44754fa62fa12c0f32f810de33f8d63db56f316d39d4329578"
Oct 30 19:31:54.750031 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:54.749925    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"723f67b7845b5f44754fa62fa12c0f32f810de33f8d63db56f316d39d4329578"} err="failed to get container status \"723f67b7845b5f44754fa62fa12c0f32f810de33f8d63db56f316d39d4329578\": rpc error: code = NotFound desc = could not find container \"723f67b7845b5f44754fa62fa12c0f32f810de33f8d63db56f316d39d4329578\": container with ID starting with 723f67b7845b5f44754fa62fa12c0f32f810de33f8d63db56f316d39d4329578 not found: ID does not exist"
Oct 30 19:31:54.758234 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:54.758199    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5e333f16-68c977c4bd-rhgmr"]
Oct 30 19:31:54.768432 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:54.768390    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5e333f16-68c977c4bd-rhgmr"]
Oct 30 19:31:55.190681 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:31:55.190635    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:31:55.767339 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:55.767281    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5e333f16-f5f6bd6dc-jfs9n" podUID="c555e0a2-316b-419d-9b37-80c44bbe9cfc" containerName="payload-pre-processing" containerID="cri-o://383d89ee213e288eafa68e0cdc577b8abd6391d40d5446d9fb0580a637138f77" gracePeriod=25
Oct 30 19:31:56.155054 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:56.155024    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5e333f16-f5f6bd6dc-jfs9n"
Oct 30 19:31:56.293673 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:56.293641    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c555e0a2-316b-419d-9b37-80c44bbe9cfc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c555e0a2-316b-419d-9b37-80c44bbe9cfc-plugins-config-volume\") pod \"c555e0a2-316b-419d-9b37-80c44bbe9cfc\" (UID: \"c555e0a2-316b-419d-9b37-80c44bbe9cfc\") "
Oct 30 19:31:56.314115 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:56.312295    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c555e0a2-316b-419d-9b37-80c44bbe9cfc-kube-api-access-9rzff\" (UniqueName: \"kubernetes.io/projected/c555e0a2-316b-419d-9b37-80c44bbe9cfc-kube-api-access-9rzff\") pod \"c555e0a2-316b-419d-9b37-80c44bbe9cfc\" (UID: \"c555e0a2-316b-419d-9b37-80c44bbe9cfc\") "
Oct 30 19:31:56.314115 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:56.311475    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c555e0a2-316b-419d-9b37-80c44bbe9cfc-plugins-config-volume" pod "c555e0a2-316b-419d-9b37-80c44bbe9cfc" (UID: "c555e0a2-316b-419d-9b37-80c44bbe9cfc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:31:56.314115 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:56.313786    5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c555e0a2-316b-419d-9b37-80c44bbe9cfc-plugins-config-volume\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:56.319020 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:56.318984    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c555e0a2-316b-419d-9b37-80c44bbe9cfc-kube-api-access-9rzff" pod "c555e0a2-316b-419d-9b37-80c44bbe9cfc" (UID: "c555e0a2-316b-419d-9b37-80c44bbe9cfc"). InnerVolumeSpecName "kube-api-access-9rzff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:31:56.416213 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:56.416110    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rzff\" (UniqueName: \"kubernetes.io/projected/c555e0a2-316b-419d-9b37-80c44bbe9cfc-kube-api-access-9rzff\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:56.704530 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:56.704436    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0669101d-0bf5-4c6c-a088-180749a12376" path="/var/lib/kubelet/pods/0669101d-0bf5-4c6c-a088-180749a12376/volumes"
Oct 30 19:31:56.705356 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:56.705303    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-229db90d-bds9s"]
Oct 30 19:31:56.705356 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:56.705351    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-229db90d-bds9s"]
Oct 30 19:31:56.842759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:56.840683    5043 generic.go:356] "Generic (PLEG): container finished" podID="c555e0a2-316b-419d-9b37-80c44bbe9cfc" containerID="383d89ee213e288eafa68e0cdc577b8abd6391d40d5446d9fb0580a637138f77" exitCode=0
Oct 30 19:31:56.842759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:56.840860    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5e333f16-f5f6bd6dc-jfs9n" event={"ID":"c555e0a2-316b-419d-9b37-80c44bbe9cfc","Type":"ContainerDied","Data":"383d89ee213e288eafa68e0cdc577b8abd6391d40d5446d9fb0580a637138f77"}
Oct 30 19:31:56.842759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:56.840942    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5e333f16-f5f6bd6dc-jfs9n" event={"ID":"c555e0a2-316b-419d-9b37-80c44bbe9cfc","Type":"ContainerDied","Data":"8f872056044465f71f94a07a72dd95c337ca6e5ce44ab13752a55b54710cbdd1"}
Oct 30 19:31:56.842759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:56.840973    5043 scope.go:122] "RemoveContainer" containerID="383d89ee213e288eafa68e0cdc577b8abd6391d40d5446d9fb0580a637138f77"
Oct 30 19:31:56.842759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:56.841139    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5e333f16-f5f6bd6dc-jfs9n"
Oct 30 19:31:56.883343 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:56.881508    5043 scope.go:122] "RemoveContainer" containerID="383d89ee213e288eafa68e0cdc577b8abd6391d40d5446d9fb0580a637138f77"
Oct 30 19:31:56.883343 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:31:56.882753    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"383d89ee213e288eafa68e0cdc577b8abd6391d40d5446d9fb0580a637138f77\": container with ID starting with 383d89ee213e288eafa68e0cdc577b8abd6391d40d5446d9fb0580a637138f77 not found: ID does not exist" containerID="383d89ee213e288eafa68e0cdc577b8abd6391d40d5446d9fb0580a637138f77"
Oct 30 19:31:56.883343 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:56.882794    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"383d89ee213e288eafa68e0cdc577b8abd6391d40d5446d9fb0580a637138f77"} err="failed to get container status \"383d89ee213e288eafa68e0cdc577b8abd6391d40d5446d9fb0580a637138f77\": rpc error: code = NotFound desc = could not find container \"383d89ee213e288eafa68e0cdc577b8abd6391d40d5446d9fb0580a637138f77\": container with ID starting with 383d89ee213e288eafa68e0cdc577b8abd6391d40d5446d9fb0580a637138f77 not found: ID does not exist"
Oct 30 19:31:56.901953 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:56.901890    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5e333f16-f5f6bd6dc-jfs9n"]
Oct 30 19:31:56.908691 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:56.908657    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5e333f16-f5f6bd6dc-jfs9n"]
Oct 30 19:31:58.649253 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:58.649128    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c555e0a2-316b-419d-9b37-80c44bbe9cfc" path="/var/lib/kubelet/pods/c555e0a2-316b-419d-9b37-80c44bbe9cfc/volumes"
Oct 30 19:31:58.650609 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:58.650581    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfc1cab2-465a-4792-a639-2a5a2ddbf6ca" path="/var/lib/kubelet/pods/dfc1cab2-465a-4792-a639-2a5a2ddbf6ca/volumes"
Oct 30 19:31:59.510906 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:59.510860    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-229db90d-6f45bf4ddb-q875f" podUID="0f7a8e0a-34a7-4421-adb6-dec8e380d134" containerName="payload-processing" containerID="cri-o://568d045dc63e0e14488784c2a2e3ee5ffb1c3cfd36cf6983e9791e2b8be94800" gracePeriod=25
Oct 30 19:31:59.724905 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:59.722825    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m"]
Oct 30 19:31:59.724905 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:59.724511    5043 cpu_manager.go:414] "containerMap: removing container" podUID="c555e0a2-316b-419d-9b37-80c44bbe9cfc" containerName="payload-pre-processing"
Oct 30 19:31:59.724905 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:59.724608    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c555e0a2-316b-419d-9b37-80c44bbe9cfc" containerName="payload-pre-processing"
Oct 30 19:31:59.724905 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:59.724804    5043 cpu_manager.go:414] "containerMap: removing container" podUID="dfc1cab2-465a-4792-a639-2a5a2ddbf6ca" containerName="revoke-keys"
Oct 30 19:31:59.724905 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:59.724818    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfc1cab2-465a-4792-a639-2a5a2ddbf6ca" containerName="revoke-keys"
Oct 30 19:31:59.724905 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:59.724865    5043 cpu_manager.go:414] "containerMap: removing container" podUID="0669101d-0bf5-4c6c-a088-180749a12376" containerName="payload-processing"
Oct 30 19:31:59.724905 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:59.724874    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0669101d-0bf5-4c6c-a088-180749a12376" containerName="payload-processing"
Oct 30 19:31:59.725659 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:59.725094    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfc1cab2-465a-4792-a639-2a5a2ddbf6ca" containerName="revoke-keys"
Oct 30 19:31:59.725659 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:59.725115    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="0669101d-0bf5-4c6c-a088-180749a12376" containerName="payload-processing"
Oct 30 19:31:59.725659 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:59.725128    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="c555e0a2-316b-419d-9b37-80c44bbe9cfc" containerName="payload-pre-processing"
Oct 30 19:31:59.742228 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:59.741141    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m"
Oct 30 19:31:59.743011 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:59.742981    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m"]
Oct 30 19:31:59.746772 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:59.746661    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-2fd9a937-openshift-default-dockercfg-wl9v2\""
Oct 30 19:31:59.807010 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:59.802503    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dzf5\" (UniqueName: \"kubernetes.io/projected/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-kube-api-access-5dzf5\") pod \"e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m\" (UID: \"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf\") " pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m"
Oct 30 19:31:59.807010 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:59.802621    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-istio-envoy\") pod \"e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m\" (UID: \"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf\") " pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m"
Oct 30 19:31:59.807010 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:59.802676    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-istiod-ca-cert\") pod \"e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m\" (UID: \"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf\") " pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m"
Oct 30 19:31:59.807010 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:59.802780    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-workload-certs\") pod \"e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m\" (UID: \"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf\") " pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m"
Oct 30 19:31:59.807010 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:59.802925    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-workload-socket\") pod \"e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m\" (UID: \"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf\") " pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m"
Oct 30 19:31:59.807010 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:59.803012    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-istio-token\") pod \"e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m\" (UID: \"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf\") " pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m"
Oct 30 19:31:59.807010 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:59.803169    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-istio-podinfo\") pod \"e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m\" (UID: \"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf\") " pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m"
Oct 30 19:31:59.807010 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:59.803240    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-credential-socket\") pod \"e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m\" (UID: \"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf\") " pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m"
Oct 30 19:31:59.807010 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:59.803273    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-istio-data\") pod \"e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m\" (UID: \"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf\") " pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m"
Oct 30 19:31:59.917771 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:59.907276    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-istio-podinfo\") pod \"e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m\" (UID: \"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf\") " pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m"
Oct 30 19:31:59.917771 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:59.907377    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-credential-socket\") pod \"e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m\" (UID: \"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf\") " pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m"
Oct 30 19:31:59.917771 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:59.907410    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-istio-data\") pod \"e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m\" (UID: \"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf\") " pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m"
Oct 30 19:31:59.917771 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:59.907470    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dzf5\" (UniqueName: \"kubernetes.io/projected/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-kube-api-access-5dzf5\") pod \"e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m\" (UID: \"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf\") " pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m"
Oct 30 19:31:59.917771 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:59.907551    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-istio-envoy\") pod \"e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m\" (UID: \"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf\") " pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m"
Oct 30 19:31:59.917771 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:59.907601    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-istiod-ca-cert\") pod \"e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m\" (UID: \"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf\") " pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m"
Oct 30 19:31:59.917771 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:59.907650    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-workload-certs\") pod \"e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m\" (UID: \"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf\") " pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m"
Oct 30 19:31:59.922761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:59.919959    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-workload-socket\") pod \"e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m\" (UID: \"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf\") " pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m"
Oct 30 19:31:59.922761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:59.920088    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-istio-token\") pod \"e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m\" (UID: \"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf\") " pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m"
Oct 30 19:31:59.974414 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:31:59.973824    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-istio-data\") pod \"e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m\" (UID: \"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf\") " pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m"
Oct 30 19:32:00.252968 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:00.192612    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-istiod-ca-cert\") pod \"e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m\" (UID: \"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf\") " pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m"
Oct 30 19:32:00.252968 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:00.192992    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-workload-socket\") pod \"e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m\" (UID: \"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf\") " pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m"
Oct 30 19:32:00.252968 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:00.193298    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-credential-socket\") pod \"e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m\" (UID: \"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf\") " pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m"
Oct 30 19:32:00.252968 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:00.232468    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-workload-certs\") pod \"e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m\" (UID: \"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf\") " pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m"
Oct 30 19:32:00.252968 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:00.233618    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dzf5\" (UniqueName: \"kubernetes.io/projected/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-kube-api-access-5dzf5\") pod \"e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m\" (UID: \"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf\") " pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m"
Oct 30 19:32:00.252968 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:00.251175    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-istio-token\") pod \"e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m\" (UID: \"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf\") " pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m"
Oct 30 19:32:00.282981 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:00.280632    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-istio-envoy\") pod \"e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m\" (UID: \"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf\") " pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m"
Oct 30 19:32:00.286242 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:00.286211    5043 generic.go:356] "Generic (PLEG): container finished" podID="0f7a8e0a-34a7-4421-adb6-dec8e380d134" containerID="568d045dc63e0e14488784c2a2e3ee5ffb1c3cfd36cf6983e9791e2b8be94800" exitCode=0
Oct 30 19:32:00.286624 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:00.286265    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-229db90d-6f45bf4ddb-q875f" event={"ID":"0f7a8e0a-34a7-4421-adb6-dec8e380d134","Type":"ContainerDied","Data":"568d045dc63e0e14488784c2a2e3ee5ffb1c3cfd36cf6983e9791e2b8be94800"}
Oct 30 19:32:00.288803 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:00.288778    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-istio-podinfo\") pod \"e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m\" (UID: \"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf\") " pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m"
Oct 30 19:32:00.309868 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:00.309841    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-229db90d-6f45bf4ddb-q875f"
Oct 30 19:32:00.498471 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:00.498442    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f7a8e0a-34a7-4421-adb6-dec8e380d134-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f7a8e0a-34a7-4421-adb6-dec8e380d134-plugins-config-volume\") pod \"0f7a8e0a-34a7-4421-adb6-dec8e380d134\" (UID: \"0f7a8e0a-34a7-4421-adb6-dec8e380d134\") "
Oct 30 19:32:00.498688 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:00.498497    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f7a8e0a-34a7-4421-adb6-dec8e380d134-kube-api-access-dzpd5\" (UniqueName: \"kubernetes.io/projected/0f7a8e0a-34a7-4421-adb6-dec8e380d134-kube-api-access-dzpd5\") pod \"0f7a8e0a-34a7-4421-adb6-dec8e380d134\" (UID: \"0f7a8e0a-34a7-4421-adb6-dec8e380d134\") "
Oct 30 19:32:00.498895 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:00.498870    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f7a8e0a-34a7-4421-adb6-dec8e380d134-plugins-config-volume" pod "0f7a8e0a-34a7-4421-adb6-dec8e380d134" (UID: "0f7a8e0a-34a7-4421-adb6-dec8e380d134"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:32:00.499221 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:00.499183    5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f7a8e0a-34a7-4421-adb6-dec8e380d134-plugins-config-volume\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:00.501067 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:00.501037    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f7a8e0a-34a7-4421-adb6-dec8e380d134-kube-api-access-dzpd5" pod "0f7a8e0a-34a7-4421-adb6-dec8e380d134" (UID: "0f7a8e0a-34a7-4421-adb6-dec8e380d134"). InnerVolumeSpecName "kube-api-access-dzpd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:32:00.503249 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:00.503199    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m"
Oct 30 19:32:00.602793 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:00.602365    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzpd5\" (UniqueName: \"kubernetes.io/projected/0f7a8e0a-34a7-4421-adb6-dec8e380d134-kube-api-access-dzpd5\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:00.830210 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:32:00.830165    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5a07d80_0678_4d80_bbc6_d17cb0ba35cf.slice/crio-63e499e88f780de67383fcb53ccc9f1f60f2b16df68fca5bcd307d3d5333a94f WatchSource:0}: Error finding container 63e499e88f780de67383fcb53ccc9f1f60f2b16df68fca5bcd307d3d5333a94f: Status 404 returned error can't find the container with id 63e499e88f780de67383fcb53ccc9f1f60f2b16df68fca5bcd307d3d5333a94f
Oct 30 19:32:00.833152 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:00.833125    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m"]
Oct 30 19:32:00.833256 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:00.833183    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:32:00.833256 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:00.833244    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:32:00.833334 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:00.833275    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:32:01.013968 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:01.013613    5043 patch_prober.go:29] interesting pod/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Oct 30 19:32:01.013968 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:01.013671    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57" podUID="e8be979e-a5c4-481a-bd2e-e7275f28c83a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Oct 30 19:32:01.452815 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:01.446108    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m" event={"ID":"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf","Type":"ContainerStarted","Data":"bc800dccc98b72bcbb1f47fb11672198a09ae67496bf6bf35a8766e88235a0f1"}
Oct 30 19:32:01.452815 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:01.446172    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m" event={"ID":"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf","Type":"ContainerStarted","Data":"63e499e88f780de67383fcb53ccc9f1f60f2b16df68fca5bcd307d3d5333a94f"}
Oct 30 19:32:01.452815 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:01.448845    5043 generic.go:356] "Generic (PLEG): container finished" podID="e8be979e-a5c4-481a-bd2e-e7275f28c83a" containerID="734b07a83502f49f5402279d544a17dd8e3a503eae4e696b948c0692b817f2df" exitCode=0
Oct 30 19:32:01.452815 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:01.448978    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57" event={"ID":"e8be979e-a5c4-481a-bd2e-e7275f28c83a","Type":"ContainerDied","Data":"734b07a83502f49f5402279d544a17dd8e3a503eae4e696b948c0692b817f2df"}
Oct 30 19:32:01.452815 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:01.449489    5043 scope.go:122] "RemoveContainer" containerID="734b07a83502f49f5402279d544a17dd8e3a503eae4e696b948c0692b817f2df"
Oct 30 19:32:01.452815 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:01.451721    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:32:01.452815 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:01.451840    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:32:01.452815 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:01.451894    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:32:01.464779 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:01.461330    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-229db90d-6f45bf4ddb-q875f" event={"ID":"0f7a8e0a-34a7-4421-adb6-dec8e380d134","Type":"ContainerDied","Data":"2ca0fefa325ec7fb633f267d0e6ddfa7ef9bfc25b1dcb311bb863dbdce09c124"}
Oct 30 19:32:01.464779 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:01.461423    5043 scope.go:122] "RemoveContainer" containerID="568d045dc63e0e14488784c2a2e3ee5ffb1c3cfd36cf6983e9791e2b8be94800"
Oct 30 19:32:01.464779 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:01.461602    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-229db90d-6f45bf4ddb-q875f"
Oct 30 19:32:01.505266 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:01.505235    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m"
Oct 30 19:32:01.505455 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:01.505427    5043 patch_prober.go:29] interesting pod/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 19:32:01.505526 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:01.505473    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m" podUID="a5a07d80-0678-4d80-bbc6-d17cb0ba35cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 19:32:01.512395 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:01.512312    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m" podStartSLOduration=2.512292787 podStartE2EDuration="2.512292787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:01.485590113 +0000 UTC m=+940.996650079" watchObservedRunningTime="2026-10-30 19:32:01.512292787 +0000 UTC m=+941.023352754"
Oct 30 19:32:01.539676 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:01.539641    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-229db90d-6f45bf4ddb-q875f"]
Oct 30 19:32:01.549330 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:01.549229    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-229db90d-6f45bf4ddb-q875f"]
Oct 30 19:32:01.641866 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:01.641836    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5e333f16-5j9n4"]
Oct 30 19:32:01.648221 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:01.648190    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5e333f16-5j9n4"]
Oct 30 19:32:01.806291 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:01.806146    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57"]
Oct 30 19:32:02.475012 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:02.474972    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57" event={"ID":"e8be979e-a5c4-481a-bd2e-e7275f28c83a","Type":"ContainerStarted","Data":"2f2ed7dd6833104fcbd733ffb46963c8fe5041d056225cf23e45ba5015c3bdc8"}
Oct 30 19:32:02.475487 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:02.475018    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57" podUID="e8be979e-a5c4-481a-bd2e-e7275f28c83a" containerName="istio-proxy" containerID="cri-o://2f2ed7dd6833104fcbd733ffb46963c8fe5041d056225cf23e45ba5015c3bdc8" gracePeriod=30
Oct 30 19:32:02.508292 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:02.508264    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m"
Oct 30 19:32:02.648856 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:02.648819    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f7a8e0a-34a7-4421-adb6-dec8e380d134" path="/var/lib/kubelet/pods/0f7a8e0a-34a7-4421-adb6-dec8e380d134/volumes"
Oct 30 19:32:02.649526 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:02.649505    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c06745d3-110e-4228-ac45-4b806095300c" path="/var/lib/kubelet/pods/c06745d3-110e-4228-ac45-4b806095300c/volumes"
Oct 30 19:32:03.489289 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:03.489260    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m"
Oct 30 19:32:03.490318 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:03.490298    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m"
Oct 30 19:32:04.343793 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:04.343759    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf"]
Oct 30 19:32:04.344506 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:04.344492    5043 cpu_manager.go:414] "containerMap: removing container" podUID="0f7a8e0a-34a7-4421-adb6-dec8e380d134" containerName="payload-processing"
Oct 30 19:32:04.344559 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:04.344507    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f7a8e0a-34a7-4421-adb6-dec8e380d134" containerName="payload-processing"
Oct 30 19:32:04.344653 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:04.344643    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f7a8e0a-34a7-4421-adb6-dec8e380d134" containerName="payload-processing"
Oct 30 19:32:04.349946 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:04.349928    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf"
Oct 30 19:32:04.363272 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:04.363246    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-ae425f-openshift-default-dockercfg-57s6q\""
Oct 30 19:32:04.368531 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:04.368499    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf"]
Oct 30 19:32:04.411675 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:04.411581    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40748d3a-8823-40ee-9425-f5e36a8889d6-workload-certs\") pod \"e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf\" (UID: \"40748d3a-8823-40ee-9425-f5e36a8889d6\") " pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf"
Oct 30 19:32:04.412076 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:04.412049    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40748d3a-8823-40ee-9425-f5e36a8889d6-istio-data\") pod \"e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf\" (UID: \"40748d3a-8823-40ee-9425-f5e36a8889d6\") " pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf"
Oct 30 19:32:04.412266 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:04.412248    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40748d3a-8823-40ee-9425-f5e36a8889d6-istio-envoy\") pod \"e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf\" (UID: \"40748d3a-8823-40ee-9425-f5e36a8889d6\") " pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf"
Oct 30 19:32:04.412423 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:04.412406    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40748d3a-8823-40ee-9425-f5e36a8889d6-istio-token\") pod \"e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf\" (UID: \"40748d3a-8823-40ee-9425-f5e36a8889d6\") " pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf"
Oct 30 19:32:04.412630 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:04.412614    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40748d3a-8823-40ee-9425-f5e36a8889d6-workload-socket\") pod \"e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf\" (UID: \"40748d3a-8823-40ee-9425-f5e36a8889d6\") " pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf"
Oct 30 19:32:04.412814 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:04.412795    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dxzs\" (UniqueName: \"kubernetes.io/projected/40748d3a-8823-40ee-9425-f5e36a8889d6-kube-api-access-9dxzs\") pod \"e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf\" (UID: \"40748d3a-8823-40ee-9425-f5e36a8889d6\") " pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf"
Oct 30 19:32:04.412998 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:04.412980    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40748d3a-8823-40ee-9425-f5e36a8889d6-istio-podinfo\") pod \"e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf\" (UID: \"40748d3a-8823-40ee-9425-f5e36a8889d6\") " pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf"
Oct 30 19:32:04.413167 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:04.413150    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40748d3a-8823-40ee-9425-f5e36a8889d6-istiod-ca-cert\") pod \"e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf\" (UID: \"40748d3a-8823-40ee-9425-f5e36a8889d6\") " pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf"
Oct 30 19:32:04.413312 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:04.413295    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40748d3a-8823-40ee-9425-f5e36a8889d6-credential-socket\") pod \"e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf\" (UID: \"40748d3a-8823-40ee-9425-f5e36a8889d6\") " pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf"
Oct 30 19:32:04.520456 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:04.520315    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40748d3a-8823-40ee-9425-f5e36a8889d6-istio-data\") pod \"e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf\" (UID: \"40748d3a-8823-40ee-9425-f5e36a8889d6\") " pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf"
Oct 30 19:32:04.523280 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:04.523254    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40748d3a-8823-40ee-9425-f5e36a8889d6-istio-envoy\") pod \"e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf\" (UID: \"40748d3a-8823-40ee-9425-f5e36a8889d6\") " pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf"
Oct 30 19:32:04.523529 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:04.523512    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40748d3a-8823-40ee-9425-f5e36a8889d6-istio-token\") pod \"e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf\" (UID: \"40748d3a-8823-40ee-9425-f5e36a8889d6\") " pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf"
Oct 30 19:32:04.524129 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:04.523161    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40748d3a-8823-40ee-9425-f5e36a8889d6-istio-data\") pod \"e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf\" (UID: \"40748d3a-8823-40ee-9425-f5e36a8889d6\") " pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf"
Oct 30 19:32:04.524391 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:04.524368    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40748d3a-8823-40ee-9425-f5e36a8889d6-workload-socket\") pod \"e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf\" (UID: \"40748d3a-8823-40ee-9425-f5e36a8889d6\") " pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf"
Oct 30 19:32:04.524542 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:04.524522    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dxzs\" (UniqueName: \"kubernetes.io/projected/40748d3a-8823-40ee-9425-f5e36a8889d6-kube-api-access-9dxzs\") pod \"e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf\" (UID: \"40748d3a-8823-40ee-9425-f5e36a8889d6\") " pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf"
Oct 30 19:32:04.524755 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:04.524721    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40748d3a-8823-40ee-9425-f5e36a8889d6-istio-podinfo\") pod \"e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf\" (UID: \"40748d3a-8823-40ee-9425-f5e36a8889d6\") " pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf"
Oct 30 19:32:04.524934 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:04.524918    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40748d3a-8823-40ee-9425-f5e36a8889d6-istiod-ca-cert\") pod \"e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf\" (UID: \"40748d3a-8823-40ee-9425-f5e36a8889d6\") " pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf"
Oct 30 19:32:04.525138 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:04.525110    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40748d3a-8823-40ee-9425-f5e36a8889d6-credential-socket\") pod \"e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf\" (UID: \"40748d3a-8823-40ee-9425-f5e36a8889d6\") " pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf"
Oct 30 19:32:04.538878 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:04.538846    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40748d3a-8823-40ee-9425-f5e36a8889d6-workload-certs\") pod \"e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf\" (UID: \"40748d3a-8823-40ee-9425-f5e36a8889d6\") " pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf"
Oct 30 19:32:04.539309 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:04.527202    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40748d3a-8823-40ee-9425-f5e36a8889d6-credential-socket\") pod \"e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf\" (UID: \"40748d3a-8823-40ee-9425-f5e36a8889d6\") " pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf"
Oct 30 19:32:04.543771 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:04.527215    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40748d3a-8823-40ee-9425-f5e36a8889d6-istiod-ca-cert\") pod \"e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf\" (UID: \"40748d3a-8823-40ee-9425-f5e36a8889d6\") " pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf"
Oct 30 19:32:04.543979 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:04.526522    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40748d3a-8823-40ee-9425-f5e36a8889d6-workload-socket\") pod \"e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf\" (UID: \"40748d3a-8823-40ee-9425-f5e36a8889d6\") " pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf"
Oct 30 19:32:04.563826 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:04.544673    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40748d3a-8823-40ee-9425-f5e36a8889d6-workload-certs\") pod \"e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf\" (UID: \"40748d3a-8823-40ee-9425-f5e36a8889d6\") " pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf"
Oct 30 19:32:04.597810 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:04.565092    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40748d3a-8823-40ee-9425-f5e36a8889d6-istio-envoy\") pod \"e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf\" (UID: \"40748d3a-8823-40ee-9425-f5e36a8889d6\") " pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf"
Oct 30 19:32:04.602767 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:04.602489    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dxzs\" (UniqueName: \"kubernetes.io/projected/40748d3a-8823-40ee-9425-f5e36a8889d6-kube-api-access-9dxzs\") pod \"e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf\" (UID: \"40748d3a-8823-40ee-9425-f5e36a8889d6\") " pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf"
Oct 30 19:32:04.613823 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:04.603483    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40748d3a-8823-40ee-9425-f5e36a8889d6-istio-podinfo\") pod \"e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf\" (UID: \"40748d3a-8823-40ee-9425-f5e36a8889d6\") " pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf"
Oct 30 19:32:04.624062 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:04.623902    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40748d3a-8823-40ee-9425-f5e36a8889d6-istio-token\") pod \"e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf\" (UID: \"40748d3a-8823-40ee-9425-f5e36a8889d6\") " pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf"
Oct 30 19:32:04.681149 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:04.681117    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf"
Oct 30 19:32:05.613273 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:05.613239    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf"]
Oct 30 19:32:05.614439 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:05.613840    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:32:05.614439 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:05.613931    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:32:05.614439 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:05.613977    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:32:06.144648 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:06.144616    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5294a7b-62g49"]
Oct 30 19:32:06.150936 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:06.150887    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5294a7b-62g49"
Oct 30 19:32:06.156904 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:06.156871    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5294a7b-62g49"]
Oct 30 19:32:06.168271 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:06.168211    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:32:06.168580 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:06.168560    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:32:06.169018 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:06.168998    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:32:06.169220 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:06.169206    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:32:06.251845 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:06.251704    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d0b40e-410b-4319-bad7-d84046f36191-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5294a7b-62g49\" (UID: \"23d0b40e-410b-4319-bad7-d84046f36191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5294a7b-62g49"
Oct 30 19:32:06.252584 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:06.251777    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjmfl\" (UniqueName: \"kubernetes.io/projected/23d0b40e-410b-4319-bad7-d84046f36191-kube-api-access-mjmfl\") pod \"maas-api-revoke-sub-e5294a7b-62g49\" (UID: \"23d0b40e-410b-4319-bad7-d84046f36191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5294a7b-62g49"
Oct 30 19:32:06.252827 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:06.252690    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d0b40e-410b-4319-bad7-d84046f36191-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5294a7b-62g49\" (UID: \"23d0b40e-410b-4319-bad7-d84046f36191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5294a7b-62g49"
Oct 30 19:32:06.332958 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:06.332914    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2fd9a937-7c795b4979-8kvvc"]
Oct 30 19:32:06.337692 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:06.337675    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2fd9a937-7c795b4979-8kvvc"
Oct 30 19:32:06.350881 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:06.350852    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2fd9a937-7c795b4979-8kvvc"]
Oct 30 19:32:06.351158 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:06.351134    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-2fd9a937\""
Oct 30 19:32:06.351273 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:06.351254    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-2fd9a937-dockercfg-llq8r\""
Oct 30 19:32:06.354401 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:06.354378    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d0b40e-410b-4319-bad7-d84046f36191-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5294a7b-62g49\" (UID: \"23d0b40e-410b-4319-bad7-d84046f36191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5294a7b-62g49"
Oct 30 19:32:06.354536 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:06.354419    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjmfl\" (UniqueName: \"kubernetes.io/projected/23d0b40e-410b-4319-bad7-d84046f36191-kube-api-access-mjmfl\") pod \"maas-api-revoke-sub-e5294a7b-62g49\" (UID: \"23d0b40e-410b-4319-bad7-d84046f36191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5294a7b-62g49"
Oct 30 19:32:06.354536 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:06.354472    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d0b40e-410b-4319-bad7-d84046f36191-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5294a7b-62g49\" (UID: \"23d0b40e-410b-4319-bad7-d84046f36191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5294a7b-62g49"
Oct 30 19:32:06.355309 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:06.355282    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d0b40e-410b-4319-bad7-d84046f36191-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5294a7b-62g49\" (UID: \"23d0b40e-410b-4319-bad7-d84046f36191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5294a7b-62g49"
Oct 30 19:32:06.357814 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:06.357791    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d0b40e-410b-4319-bad7-d84046f36191-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5294a7b-62g49\" (UID: \"23d0b40e-410b-4319-bad7-d84046f36191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5294a7b-62g49"
Oct 30 19:32:06.371059 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:06.371029    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjmfl\" (UniqueName: \"kubernetes.io/projected/23d0b40e-410b-4319-bad7-d84046f36191-kube-api-access-mjmfl\") pod \"maas-api-revoke-sub-e5294a7b-62g49\" (UID: \"23d0b40e-410b-4319-bad7-d84046f36191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5294a7b-62g49"
Oct 30 19:32:06.460811 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:06.458986    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01e9b019-f11b-4fc9-9044-106429eddcfd-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2fd9a937-7c795b4979-8kvvc\" (UID: \"01e9b019-f11b-4fc9-9044-106429eddcfd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2fd9a937-7c795b4979-8kvvc"
Oct 30 19:32:06.460811 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:06.459125    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwn2x\" (UniqueName: \"kubernetes.io/projected/01e9b019-f11b-4fc9-9044-106429eddcfd-kube-api-access-rwn2x\") pod \"payload-pre-processing-e2e-mt-2fd9a937-7c795b4979-8kvvc\" (UID: \"01e9b019-f11b-4fc9-9044-106429eddcfd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2fd9a937-7c795b4979-8kvvc"
Oct 30 19:32:06.492786 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:06.487019    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5294a7b-62g49"
Oct 30 19:32:06.588036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:06.582283    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01e9b019-f11b-4fc9-9044-106429eddcfd-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2fd9a937-7c795b4979-8kvvc\" (UID: \"01e9b019-f11b-4fc9-9044-106429eddcfd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2fd9a937-7c795b4979-8kvvc"
Oct 30 19:32:06.588036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:06.582426    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwn2x\" (UniqueName: \"kubernetes.io/projected/01e9b019-f11b-4fc9-9044-106429eddcfd-kube-api-access-rwn2x\") pod \"payload-pre-processing-e2e-mt-2fd9a937-7c795b4979-8kvvc\" (UID: \"01e9b019-f11b-4fc9-9044-106429eddcfd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2fd9a937-7c795b4979-8kvvc"
Oct 30 19:32:06.589697 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:06.589662    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01e9b019-f11b-4fc9-9044-106429eddcfd-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2fd9a937-7c795b4979-8kvvc\" (UID: \"01e9b019-f11b-4fc9-9044-106429eddcfd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2fd9a937-7c795b4979-8kvvc"
Oct 30 19:32:06.679764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:06.655574    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwn2x\" (UniqueName: \"kubernetes.io/projected/01e9b019-f11b-4fc9-9044-106429eddcfd-kube-api-access-rwn2x\") pod \"payload-pre-processing-e2e-mt-2fd9a937-7c795b4979-8kvvc\" (UID: \"01e9b019-f11b-4fc9-9044-106429eddcfd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2fd9a937-7c795b4979-8kvvc"
Oct 30 19:32:06.706758 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:06.706557    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf" event={"ID":"40748d3a-8823-40ee-9425-f5e36a8889d6","Type":"ContainerStarted","Data":"90c4e9cd64ebdd562a5dc406f8343f7b669002370aa53934d44370f6fcf30262"}
Oct 30 19:32:06.706758 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:06.706611    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf" event={"ID":"40748d3a-8823-40ee-9425-f5e36a8889d6","Type":"ContainerStarted","Data":"d7d229d468c17396abf169c27db7a18d9eea44ddf701899963d25fc37385d84c"}
Oct 30 19:32:06.711851 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:06.708447    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf"
Oct 30 19:32:06.718495 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:06.718463    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf"
Oct 30 19:32:06.730261 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:06.730209    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf" podStartSLOduration=2.730193237 podStartE2EDuration="2.730193237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:06.727338153 +0000 UTC m=+946.238398117" watchObservedRunningTime="2026-10-30 19:32:06.730193237 +0000 UTC m=+946.241253203"
Oct 30 19:32:06.964104 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:06.964014    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2fd9a937-7c795b4979-8kvvc"
Oct 30 19:32:07.015802 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:07.012813    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5294a7b-62g49"]
Oct 30 19:32:07.306299 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:32:07.306266    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01e9b019_f11b_4fc9_9044_106429eddcfd.slice/crio-3925e0536a2083d7bfd3280808a9657261abd07415fd698e1bfaebdd78dbe621 WatchSource:0}: Error finding container 3925e0536a2083d7bfd3280808a9657261abd07415fd698e1bfaebdd78dbe621: Status 404 returned error can't find the container with id 3925e0536a2083d7bfd3280808a9657261abd07415fd698e1bfaebdd78dbe621
Oct 30 19:32:07.308474 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:07.308418    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2fd9a937-7c795b4979-8kvvc"]
Oct 30 19:32:07.604944 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:07.604915    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5cb1c5cd-5tq7t"]
Oct 30 19:32:07.610610 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:07.610586    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5cb1c5cd-5tq7t"
Oct 30 19:32:07.618110 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:07.618086    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5cb1c5cd-5tq7t"]
Oct 30 19:32:07.685844 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:07.684614    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v96n4\" (UniqueName: \"kubernetes.io/projected/fdd1e39c-3a49-4153-ab0f-e61d1ce33839-kube-api-access-v96n4\") pod \"maas-api-revoke-keys-e2e-derive-5cb1c5cd-5tq7t\" (UID: \"fdd1e39c-3a49-4153-ab0f-e61d1ce33839\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5cb1c5cd-5tq7t"
Oct 30 19:32:07.685844 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:07.684902    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdd1e39c-3a49-4153-ab0f-e61d1ce33839-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-5cb1c5cd-5tq7t\" (UID: \"fdd1e39c-3a49-4153-ab0f-e61d1ce33839\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5cb1c5cd-5tq7t"
Oct 30 19:32:07.685844 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:07.685016    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdd1e39c-3a49-4153-ab0f-e61d1ce33839-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-5cb1c5cd-5tq7t\" (UID: \"fdd1e39c-3a49-4153-ab0f-e61d1ce33839\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5cb1c5cd-5tq7t"
Oct 30 19:32:07.720203 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:07.720163    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2fd9a937-7c795b4979-8kvvc" event={"ID":"01e9b019-f11b-4fc9-9044-106429eddcfd","Type":"ContainerStarted","Data":"7722105df0dd6143efe99dfd33200b95b3df1a87432ff15647463ea61535bb1d"}
Oct 30 19:32:07.720479 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:07.720462    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2fd9a937-7c795b4979-8kvvc" event={"ID":"01e9b019-f11b-4fc9-9044-106429eddcfd","Type":"ContainerStarted","Data":"3925e0536a2083d7bfd3280808a9657261abd07415fd698e1bfaebdd78dbe621"}
Oct 30 19:32:07.722307 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:07.722285    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2fd9a937-7c795b4979-8kvvc"
Oct 30 19:32:07.725317 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:07.725280    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5294a7b-62g49" event={"ID":"23d0b40e-410b-4319-bad7-d84046f36191","Type":"ContainerStarted","Data":"e5a67b1d6e4ebd84400f9709701fa2befa9c18f1a355626c667f8d29c74a5698"}
Oct 30 19:32:07.725438 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:07.725350    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5294a7b-62g49" event={"ID":"23d0b40e-410b-4319-bad7-d84046f36191","Type":"ContainerStarted","Data":"5d11c03ce0862a537db1e057ad50e29d60083210355e56e0e466158b3abd97f5"}
Oct 30 19:32:07.729310 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:07.729288    5043 generic.go:356] "Generic (PLEG): container finished" podID="e8be979e-a5c4-481a-bd2e-e7275f28c83a" containerID="2f2ed7dd6833104fcbd733ffb46963c8fe5041d056225cf23e45ba5015c3bdc8" exitCode=0
Oct 30 19:32:07.729723 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:07.729680    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57" event={"ID":"e8be979e-a5c4-481a-bd2e-e7275f28c83a","Type":"ContainerDied","Data":"2f2ed7dd6833104fcbd733ffb46963c8fe5041d056225cf23e45ba5015c3bdc8"}
Oct 30 19:32:07.729851 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:07.729778    5043 scope.go:122] "RemoveContainer" containerID="734b07a83502f49f5402279d544a17dd8e3a503eae4e696b948c0692b817f2df"
Oct 30 19:32:07.730367 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:07.730347    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf"
Oct 30 19:32:07.731425 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:07.731321    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf"
Oct 30 19:32:07.748021 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:07.747094    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-2fd9a937-7c795b4979-8kvvc" podStartSLOduration=1.7470779109999999 podStartE2EDuration="1.747077911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:07.7400419 +0000 UTC m=+947.251101866" watchObservedRunningTime="2026-10-30 19:32:07.747077911 +0000 UTC m=+947.258137877"
Oct 30 19:32:07.788439 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:07.788411    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v96n4\" (UniqueName: \"kubernetes.io/projected/fdd1e39c-3a49-4153-ab0f-e61d1ce33839-kube-api-access-v96n4\") pod \"maas-api-revoke-keys-e2e-derive-5cb1c5cd-5tq7t\" (UID: \"fdd1e39c-3a49-4153-ab0f-e61d1ce33839\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5cb1c5cd-5tq7t"
Oct 30 19:32:07.788923 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:07.788882    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdd1e39c-3a49-4153-ab0f-e61d1ce33839-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-5cb1c5cd-5tq7t\" (UID: \"fdd1e39c-3a49-4153-ab0f-e61d1ce33839\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5cb1c5cd-5tq7t"
Oct 30 19:32:07.789271 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:07.789252    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdd1e39c-3a49-4153-ab0f-e61d1ce33839-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-5cb1c5cd-5tq7t\" (UID: \"fdd1e39c-3a49-4153-ab0f-e61d1ce33839\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5cb1c5cd-5tq7t"
Oct 30 19:32:07.795639 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:07.790366    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdd1e39c-3a49-4153-ab0f-e61d1ce33839-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-5cb1c5cd-5tq7t\" (UID: \"fdd1e39c-3a49-4153-ab0f-e61d1ce33839\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5cb1c5cd-5tq7t"
Oct 30 19:32:07.802039 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:07.801909    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5294a7b-62g49" podStartSLOduration=1.801891997 podStartE2EDuration="1.801891997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:07.767509205 +0000 UTC m=+947.278569174" watchObservedRunningTime="2026-10-30 19:32:07.801891997 +0000 UTC m=+947.312951966"
Oct 30 19:32:07.819214 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:07.819127    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdd1e39c-3a49-4153-ab0f-e61d1ce33839-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-5cb1c5cd-5tq7t\" (UID: \"fdd1e39c-3a49-4153-ab0f-e61d1ce33839\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5cb1c5cd-5tq7t"
Oct 30 19:32:07.819701 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:07.819678    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v96n4\" (UniqueName: \"kubernetes.io/projected/fdd1e39c-3a49-4153-ab0f-e61d1ce33839-kube-api-access-v96n4\") pod \"maas-api-revoke-keys-e2e-derive-5cb1c5cd-5tq7t\" (UID: \"fdd1e39c-3a49-4153-ab0f-e61d1ce33839\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5cb1c5cd-5tq7t"
Oct 30 19:32:07.898687 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:07.898654    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57"
Oct 30 19:32:07.945213 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:07.945165    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5cb1c5cd-5tq7t"
Oct 30 19:32:08.001248 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:08.001197    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8be979e-a5c4-481a-bd2e-e7275f28c83a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8be979e-a5c4-481a-bd2e-e7275f28c83a-workload-certs\") pod \"e8be979e-a5c4-481a-bd2e-e7275f28c83a\" (UID: \"e8be979e-a5c4-481a-bd2e-e7275f28c83a\") "
Oct 30 19:32:08.001381 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:08.001289    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8be979e-a5c4-481a-bd2e-e7275f28c83a-istio-token\" (UniqueName: \"kubernetes.io/projected/e8be979e-a5c4-481a-bd2e-e7275f28c83a-istio-token\") pod \"e8be979e-a5c4-481a-bd2e-e7275f28c83a\" (UID: \"e8be979e-a5c4-481a-bd2e-e7275f28c83a\") "
Oct 30 19:32:08.001484 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:08.001445    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8be979e-a5c4-481a-bd2e-e7275f28c83a-kube-api-access-fjpp6\" (UniqueName: \"kubernetes.io/projected/e8be979e-a5c4-481a-bd2e-e7275f28c83a-kube-api-access-fjpp6\") pod \"e8be979e-a5c4-481a-bd2e-e7275f28c83a\" (UID: \"e8be979e-a5c4-481a-bd2e-e7275f28c83a\") "
Oct 30 19:32:08.002030 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:08.002006    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8be979e-a5c4-481a-bd2e-e7275f28c83a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8be979e-a5c4-481a-bd2e-e7275f28c83a-credential-socket\") pod \"e8be979e-a5c4-481a-bd2e-e7275f28c83a\" (UID: \"e8be979e-a5c4-481a-bd2e-e7275f28c83a\") "
Oct 30 19:32:08.002101 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:08.002085    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8be979e-a5c4-481a-bd2e-e7275f28c83a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8be979e-a5c4-481a-bd2e-e7275f28c83a-istio-data\") pod \"e8be979e-a5c4-481a-bd2e-e7275f28c83a\" (UID: \"e8be979e-a5c4-481a-bd2e-e7275f28c83a\") "
Oct 30 19:32:08.002182 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:08.002166    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8be979e-a5c4-481a-bd2e-e7275f28c83a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8be979e-a5c4-481a-bd2e-e7275f28c83a-istio-envoy\") pod \"e8be979e-a5c4-481a-bd2e-e7275f28c83a\" (UID: \"e8be979e-a5c4-481a-bd2e-e7275f28c83a\") "
Oct 30 19:32:08.002277 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:08.002260    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8be979e-a5c4-481a-bd2e-e7275f28c83a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8be979e-a5c4-481a-bd2e-e7275f28c83a-istio-podinfo\") pod \"e8be979e-a5c4-481a-bd2e-e7275f28c83a\" (UID: \"e8be979e-a5c4-481a-bd2e-e7275f28c83a\") "
Oct 30 19:32:08.002406 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:08.002392    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8be979e-a5c4-481a-bd2e-e7275f28c83a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8be979e-a5c4-481a-bd2e-e7275f28c83a-istiod-ca-cert\") pod \"e8be979e-a5c4-481a-bd2e-e7275f28c83a\" (UID: \"e8be979e-a5c4-481a-bd2e-e7275f28c83a\") "
Oct 30 19:32:08.002494 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:08.002481    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8be979e-a5c4-481a-bd2e-e7275f28c83a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8be979e-a5c4-481a-bd2e-e7275f28c83a-workload-socket\") pod \"e8be979e-a5c4-481a-bd2e-e7275f28c83a\" (UID: \"e8be979e-a5c4-481a-bd2e-e7275f28c83a\") "
Oct 30 19:32:08.003749 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:08.003708    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8be979e-a5c4-481a-bd2e-e7275f28c83a-workload-socket" pod "e8be979e-a5c4-481a-bd2e-e7275f28c83a" (UID: "e8be979e-a5c4-481a-bd2e-e7275f28c83a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:32:08.004023 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:08.004003    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8be979e-a5c4-481a-bd2e-e7275f28c83a-workload-certs" pod "e8be979e-a5c4-481a-bd2e-e7275f28c83a" (UID: "e8be979e-a5c4-481a-bd2e-e7275f28c83a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:32:08.005879 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:08.004935    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8be979e-a5c4-481a-bd2e-e7275f28c83a-istio-data" pod "e8be979e-a5c4-481a-bd2e-e7275f28c83a" (UID: "e8be979e-a5c4-481a-bd2e-e7275f28c83a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:32:08.005879 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:08.005064    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8be979e-a5c4-481a-bd2e-e7275f28c83a-credential-socket" pod "e8be979e-a5c4-481a-bd2e-e7275f28c83a" (UID: "e8be979e-a5c4-481a-bd2e-e7275f28c83a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:32:08.005879 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:08.005363    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8be979e-a5c4-481a-bd2e-e7275f28c83a-istiod-ca-cert" pod "e8be979e-a5c4-481a-bd2e-e7275f28c83a" (UID: "e8be979e-a5c4-481a-bd2e-e7275f28c83a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:32:08.012379 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:08.012344    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8be979e-a5c4-481a-bd2e-e7275f28c83a-istio-envoy" pod "e8be979e-a5c4-481a-bd2e-e7275f28c83a" (UID: "e8be979e-a5c4-481a-bd2e-e7275f28c83a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:32:08.013106 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:08.013072    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8be979e-a5c4-481a-bd2e-e7275f28c83a-kube-api-access-fjpp6" pod "e8be979e-a5c4-481a-bd2e-e7275f28c83a" (UID: "e8be979e-a5c4-481a-bd2e-e7275f28c83a"). InnerVolumeSpecName "kube-api-access-fjpp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:32:08.013252 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:08.013225    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8be979e-a5c4-481a-bd2e-e7275f28c83a-istio-podinfo" pod "e8be979e-a5c4-481a-bd2e-e7275f28c83a" (UID: "e8be979e-a5c4-481a-bd2e-e7275f28c83a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:32:08.021081 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:08.021026    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8be979e-a5c4-481a-bd2e-e7275f28c83a-istio-token" pod "e8be979e-a5c4-481a-bd2e-e7275f28c83a" (UID: "e8be979e-a5c4-481a-bd2e-e7275f28c83a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:32:08.107191 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:08.107158    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8be979e-a5c4-481a-bd2e-e7275f28c83a-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:08.107191 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:08.107192    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8be979e-a5c4-481a-bd2e-e7275f28c83a-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:08.107406 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:08.107206    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8be979e-a5c4-481a-bd2e-e7275f28c83a-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:08.107406 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:08.107220    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8be979e-a5c4-481a-bd2e-e7275f28c83a-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:08.107406 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:08.107236    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8be979e-a5c4-481a-bd2e-e7275f28c83a-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:08.107406 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:08.107250    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8be979e-a5c4-481a-bd2e-e7275f28c83a-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:08.107406 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:08.107259    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8be979e-a5c4-481a-bd2e-e7275f28c83a-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:08.107406 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:08.107268    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8be979e-a5c4-481a-bd2e-e7275f28c83a-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:08.107406 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:08.107278    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjpp6\" (UniqueName: \"kubernetes.io/projected/e8be979e-a5c4-481a-bd2e-e7275f28c83a-kube-api-access-fjpp6\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:08.338498 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:32:08.338466    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdd1e39c_3a49_4153_ab0f_e61d1ce33839.slice/crio-2df9a41af23828ba90386638b59d94e4a39fc1911dbb86df3d61f25f01c4429b WatchSource:0}: Error finding container 2df9a41af23828ba90386638b59d94e4a39fc1911dbb86df3d61f25f01c4429b: Status 404 returned error can't find the container with id 2df9a41af23828ba90386638b59d94e4a39fc1911dbb86df3d61f25f01c4429b
Oct 30 19:32:08.339864 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:08.339839    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5cb1c5cd-5tq7t"]
Oct 30 19:32:08.743156 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:08.743103    5043 generic.go:356] "Generic (PLEG): container finished" podID="23d0b40e-410b-4319-bad7-d84046f36191" containerID="e5a67b1d6e4ebd84400f9709701fa2befa9c18f1a355626c667f8d29c74a5698" exitCode=0
Oct 30 19:32:08.743621 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:08.743250    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5294a7b-62g49" event={"ID":"23d0b40e-410b-4319-bad7-d84046f36191","Type":"ContainerDied","Data":"e5a67b1d6e4ebd84400f9709701fa2befa9c18f1a355626c667f8d29c74a5698"}
Oct 30 19:32:08.745239 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:08.745216    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57"
Oct 30 19:32:08.745419 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:08.745280    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57" event={"ID":"e8be979e-a5c4-481a-bd2e-e7275f28c83a","Type":"ContainerDied","Data":"a5be4325bd1791999046816d93e5956606759de11fe2f9135f4cb04a1f0b078d"}
Oct 30 19:32:08.745419 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:08.745367    5043 scope.go:122] "RemoveContainer" containerID="2f2ed7dd6833104fcbd733ffb46963c8fe5041d056225cf23e45ba5015c3bdc8"
Oct 30 19:32:08.747609 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:08.747372    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5cb1c5cd-5tq7t" event={"ID":"fdd1e39c-3a49-4153-ab0f-e61d1ce33839","Type":"ContainerStarted","Data":"2df9a41af23828ba90386638b59d94e4a39fc1911dbb86df3d61f25f01c4429b"}
Oct 30 19:32:08.787584 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:08.787546    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57"]
Oct 30 19:32:08.795427 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:08.795393    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-5e333f16-gw-openshift-default-7c796c78f-kpx57"]
Oct 30 19:32:09.761443 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:09.761406    5043 generic.go:356] "Generic (PLEG): container finished" podID="fdd1e39c-3a49-4153-ab0f-e61d1ce33839" containerID="3af35e38edcbc548d2ab34f0c064ae1232d89a9d115d6e10fd0e3ec1cfa1611c" exitCode=0
Oct 30 19:32:09.761899 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:09.761499    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5cb1c5cd-5tq7t" event={"ID":"fdd1e39c-3a49-4153-ab0f-e61d1ce33839","Type":"ContainerDied","Data":"3af35e38edcbc548d2ab34f0c064ae1232d89a9d115d6e10fd0e3ec1cfa1611c"}
Oct 30 19:32:09.968872 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:09.968843    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5294a7b-62g49"
Oct 30 19:32:09.999226 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:09.999190    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23d0b40e-410b-4319-bad7-d84046f36191-kube-api-access-mjmfl\" (UniqueName: \"kubernetes.io/projected/23d0b40e-410b-4319-bad7-d84046f36191-kube-api-access-mjmfl\") pod \"23d0b40e-410b-4319-bad7-d84046f36191\" (UID: \"23d0b40e-410b-4319-bad7-d84046f36191\") "
Oct 30 19:32:09.999472 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:09.999454    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23d0b40e-410b-4319-bad7-d84046f36191-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d0b40e-410b-4319-bad7-d84046f36191-maas-api-openshift-ca\") pod \"23d0b40e-410b-4319-bad7-d84046f36191\" (UID: \"23d0b40e-410b-4319-bad7-d84046f36191\") "
Oct 30 19:32:09.999561 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:09.999545    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23d0b40e-410b-4319-bad7-d84046f36191-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d0b40e-410b-4319-bad7-d84046f36191-maas-api-xks-ca\") pod \"23d0b40e-410b-4319-bad7-d84046f36191\" (UID: \"23d0b40e-410b-4319-bad7-d84046f36191\") "
Oct 30 19:32:10.001915 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:10.001885    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23d0b40e-410b-4319-bad7-d84046f36191-maas-api-openshift-ca" pod "23d0b40e-410b-4319-bad7-d84046f36191" (UID: "23d0b40e-410b-4319-bad7-d84046f36191"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:32:10.007374 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:10.007339    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d0b40e-410b-4319-bad7-d84046f36191-kube-api-access-mjmfl" pod "23d0b40e-410b-4319-bad7-d84046f36191" (UID: "23d0b40e-410b-4319-bad7-d84046f36191"). InnerVolumeSpecName "kube-api-access-mjmfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:32:10.014649 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:10.014622    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23d0b40e-410b-4319-bad7-d84046f36191-maas-api-xks-ca" pod "23d0b40e-410b-4319-bad7-d84046f36191" (UID: "23d0b40e-410b-4319-bad7-d84046f36191"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:32:10.101648 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:10.101616    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d0b40e-410b-4319-bad7-d84046f36191-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:10.101648 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:10.101645    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d0b40e-410b-4319-bad7-d84046f36191-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:10.101648 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:10.101656    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjmfl\" (UniqueName: \"kubernetes.io/projected/23d0b40e-410b-4319-bad7-d84046f36191-kube-api-access-mjmfl\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:10.570054 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:10.570013    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ae425f-79c9ff7db7-z7nf7"]
Oct 30 19:32:10.571878 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:10.571766    5043 cpu_manager.go:414] "containerMap: removing container" podUID="e8be979e-a5c4-481a-bd2e-e7275f28c83a" containerName="istio-proxy"
Oct 30 19:32:10.571878 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:10.571792    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8be979e-a5c4-481a-bd2e-e7275f28c83a" containerName="istio-proxy"
Oct 30 19:32:10.571878 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:10.571853    5043 cpu_manager.go:414] "containerMap: removing container" podUID="23d0b40e-410b-4319-bad7-d84046f36191" containerName="revoke-keys"
Oct 30 19:32:10.572099 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:10.571890    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23d0b40e-410b-4319-bad7-d84046f36191" containerName="revoke-keys"
Oct 30 19:32:10.572099 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:10.571931    5043 cpu_manager.go:414] "containerMap: removing container" podUID="e8be979e-a5c4-481a-bd2e-e7275f28c83a" containerName="istio-proxy"
Oct 30 19:32:10.572099 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:10.571940    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8be979e-a5c4-481a-bd2e-e7275f28c83a" containerName="istio-proxy"
Oct 30 19:32:10.572339 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:10.572319    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8be979e-a5c4-481a-bd2e-e7275f28c83a" containerName="istio-proxy"
Oct 30 19:32:10.572405 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:10.572356    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="23d0b40e-410b-4319-bad7-d84046f36191" containerName="revoke-keys"
Oct 30 19:32:10.579265 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:10.579239    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ae425f-79c9ff7db7-z7nf7"
Oct 30 19:32:10.581848 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:10.581822    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ae425f-79c9ff7db7-z7nf7"]
Oct 30 19:32:10.586145 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:10.586115    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-ae425f-dockercfg-78tzc\""
Oct 30 19:32:10.591480 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:10.591460    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-ae425f\""
Oct 30 19:32:10.605480 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:10.605448    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-ae425f-5885c86dd4-4wvh9"]
Oct 30 19:32:10.606351 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:10.606330    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8be979e-a5c4-481a-bd2e-e7275f28c83a" containerName="istio-proxy"
Oct 30 19:32:10.610812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:10.610792    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ae425f-5885c86dd4-4wvh9"
Oct 30 19:32:10.619689 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:10.619480    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7753bd1c-f916-40db-a923-eb46e92a548e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-ae425f-79c9ff7db7-z7nf7\" (UID: \"7753bd1c-f916-40db-a923-eb46e92a548e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ae425f-79c9ff7db7-z7nf7"
Oct 30 19:32:10.619902 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:10.619864    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5n7w\" (UniqueName: \"kubernetes.io/projected/7753bd1c-f916-40db-a923-eb46e92a548e-kube-api-access-q5n7w\") pod \"payload-pre-processing-e2e-shared-b-w0-ae425f-79c9ff7db7-z7nf7\" (UID: \"7753bd1c-f916-40db-a923-eb46e92a548e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ae425f-79c9ff7db7-z7nf7"
Oct 30 19:32:10.628763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:10.628719    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-ae425f-5885c86dd4-4wvh9"]
Oct 30 19:32:10.759381 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:10.759351    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/469e981b-1193-4d89-b98a-bc6f794cb532-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-ae425f-5885c86dd4-4wvh9\" (UID: \"469e981b-1193-4d89-b98a-bc6f794cb532\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ae425f-5885c86dd4-4wvh9"
Oct 30 19:32:10.759826 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:10.759795    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7753bd1c-f916-40db-a923-eb46e92a548e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-ae425f-79c9ff7db7-z7nf7\" (UID: \"7753bd1c-f916-40db-a923-eb46e92a548e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ae425f-79c9ff7db7-z7nf7"
Oct 30 19:32:10.760058 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:10.760038    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5n7w\" (UniqueName: \"kubernetes.io/projected/7753bd1c-f916-40db-a923-eb46e92a548e-kube-api-access-q5n7w\") pod \"payload-pre-processing-e2e-shared-b-w0-ae425f-79c9ff7db7-z7nf7\" (UID: \"7753bd1c-f916-40db-a923-eb46e92a548e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ae425f-79c9ff7db7-z7nf7"
Oct 30 19:32:10.760371 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:10.760353    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h86w\" (UniqueName: \"kubernetes.io/projected/469e981b-1193-4d89-b98a-bc6f794cb532-kube-api-access-4h86w\") pod \"payload-processing-e2e-shared-b-w0-ae425f-5885c86dd4-4wvh9\" (UID: \"469e981b-1193-4d89-b98a-bc6f794cb532\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ae425f-5885c86dd4-4wvh9"
Oct 30 19:32:10.762038 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:10.762016    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7753bd1c-f916-40db-a923-eb46e92a548e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-ae425f-79c9ff7db7-z7nf7\" (UID: \"7753bd1c-f916-40db-a923-eb46e92a548e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ae425f-79c9ff7db7-z7nf7"
Oct 30 19:32:10.936163 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:10.936011    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h86w\" (UniqueName: \"kubernetes.io/projected/469e981b-1193-4d89-b98a-bc6f794cb532-kube-api-access-4h86w\") pod \"payload-processing-e2e-shared-b-w0-ae425f-5885c86dd4-4wvh9\" (UID: \"469e981b-1193-4d89-b98a-bc6f794cb532\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ae425f-5885c86dd4-4wvh9"
Oct 30 19:32:10.936593 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:10.936568    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/469e981b-1193-4d89-b98a-bc6f794cb532-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-ae425f-5885c86dd4-4wvh9\" (UID: \"469e981b-1193-4d89-b98a-bc6f794cb532\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ae425f-5885c86dd4-4wvh9"
Oct 30 19:32:10.937767 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:10.937722    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/469e981b-1193-4d89-b98a-bc6f794cb532-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-ae425f-5885c86dd4-4wvh9\" (UID: \"469e981b-1193-4d89-b98a-bc6f794cb532\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ae425f-5885c86dd4-4wvh9"
Oct 30 19:32:10.997862 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:10.997818    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5n7w\" (UniqueName: \"kubernetes.io/projected/7753bd1c-f916-40db-a923-eb46e92a548e-kube-api-access-q5n7w\") pod \"payload-pre-processing-e2e-shared-b-w0-ae425f-79c9ff7db7-z7nf7\" (UID: \"7753bd1c-f916-40db-a923-eb46e92a548e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ae425f-79c9ff7db7-z7nf7"
Oct 30 19:32:11.002522 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:11.002490    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h86w\" (UniqueName: \"kubernetes.io/projected/469e981b-1193-4d89-b98a-bc6f794cb532-kube-api-access-4h86w\") pod \"payload-processing-e2e-shared-b-w0-ae425f-5885c86dd4-4wvh9\" (UID: \"469e981b-1193-4d89-b98a-bc6f794cb532\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ae425f-5885c86dd4-4wvh9"
Oct 30 19:32:11.006482 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:11.006424    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8be979e-a5c4-481a-bd2e-e7275f28c83a" path="/var/lib/kubelet/pods/e8be979e-a5c4-481a-bd2e-e7275f28c83a/volumes"
Oct 30 19:32:11.041993 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:11.019357    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5294a7b-62g49"
Oct 30 19:32:11.049765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:11.043272    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5294a7b-62g49" event={"ID":"23d0b40e-410b-4319-bad7-d84046f36191","Type":"ContainerDied","Data":"5d11c03ce0862a537db1e057ad50e29d60083210355e56e0e466158b3abd97f5"}
Oct 30 19:32:11.049765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:11.043430    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d11c03ce0862a537db1e057ad50e29d60083210355e56e0e466158b3abd97f5"
Oct 30 19:32:11.199020 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:11.198942    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ae425f-79c9ff7db7-z7nf7"
Oct 30 19:32:11.228383 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:11.228340    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ae425f-5885c86dd4-4wvh9"
Oct 30 19:32:11.271604 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:11.271568    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5cb1c5cd-5tq7t"
Oct 30 19:32:11.377992 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:11.377270    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdd1e39c-3a49-4153-ab0f-e61d1ce33839-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdd1e39c-3a49-4153-ab0f-e61d1ce33839-maas-api-xks-ca\") pod \"fdd1e39c-3a49-4153-ab0f-e61d1ce33839\" (UID: \"fdd1e39c-3a49-4153-ab0f-e61d1ce33839\") "
Oct 30 19:32:11.377992 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:11.377430    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdd1e39c-3a49-4153-ab0f-e61d1ce33839-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdd1e39c-3a49-4153-ab0f-e61d1ce33839-maas-api-openshift-ca\") pod \"fdd1e39c-3a49-4153-ab0f-e61d1ce33839\" (UID: \"fdd1e39c-3a49-4153-ab0f-e61d1ce33839\") "
Oct 30 19:32:11.377992 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:11.377572    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdd1e39c-3a49-4153-ab0f-e61d1ce33839-kube-api-access-v96n4\" (UniqueName: \"kubernetes.io/projected/fdd1e39c-3a49-4153-ab0f-e61d1ce33839-kube-api-access-v96n4\") pod \"fdd1e39c-3a49-4153-ab0f-e61d1ce33839\" (UID: \"fdd1e39c-3a49-4153-ab0f-e61d1ce33839\") "
Oct 30 19:32:11.394711 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:11.381494    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdd1e39c-3a49-4153-ab0f-e61d1ce33839-maas-api-openshift-ca" pod "fdd1e39c-3a49-4153-ab0f-e61d1ce33839" (UID: "fdd1e39c-3a49-4153-ab0f-e61d1ce33839"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:32:11.394711 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:11.391624    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdd1e39c-3a49-4153-ab0f-e61d1ce33839-maas-api-xks-ca" pod "fdd1e39c-3a49-4153-ab0f-e61d1ce33839" (UID: "fdd1e39c-3a49-4153-ab0f-e61d1ce33839"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:32:11.397021 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:11.396287    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdd1e39c-3a49-4153-ab0f-e61d1ce33839-kube-api-access-v96n4" pod "fdd1e39c-3a49-4153-ab0f-e61d1ce33839" (UID: "fdd1e39c-3a49-4153-ab0f-e61d1ce33839"). InnerVolumeSpecName "kube-api-access-v96n4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:32:11.480436 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:11.480395    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdd1e39c-3a49-4153-ab0f-e61d1ce33839-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:11.480436 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:11.480433    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdd1e39c-3a49-4153-ab0f-e61d1ce33839-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:11.480635 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:11.480449    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v96n4\" (UniqueName: \"kubernetes.io/projected/fdd1e39c-3a49-4153-ab0f-e61d1ce33839-kube-api-access-v96n4\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:11.558411 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:32:11.558378    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7753bd1c_f916_40db_a923_eb46e92a548e.slice/crio-d6de27d69c4218f657a24814d652c79dd780e72cca15159ef63314e07464756a WatchSource:0}: Error finding container d6de27d69c4218f657a24814d652c79dd780e72cca15159ef63314e07464756a: Status 404 returned error can't find the container with id d6de27d69c4218f657a24814d652c79dd780e72cca15159ef63314e07464756a
Oct 30 19:32:11.561991 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:11.561943    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ae425f-79c9ff7db7-z7nf7"]
Oct 30 19:32:11.603201 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:32:11.603168    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod469e981b_1193_4d89_b98a_bc6f794cb532.slice/crio-7bd85e8380caa5a3b3e981fe3eddd8a500b32f9d1512f84e6cc6cab8e4ad97d2 WatchSource:0}: Error finding container 7bd85e8380caa5a3b3e981fe3eddd8a500b32f9d1512f84e6cc6cab8e4ad97d2: Status 404 returned error can't find the container with id 7bd85e8380caa5a3b3e981fe3eddd8a500b32f9d1512f84e6cc6cab8e4ad97d2
Oct 30 19:32:11.604944 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:11.604916    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-ae425f-5885c86dd4-4wvh9"]
Oct 30 19:32:11.972301 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:11.972255    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ae425f-79c9ff7db7-z7nf7" event={"ID":"7753bd1c-f916-40db-a923-eb46e92a548e","Type":"ContainerStarted","Data":"c32fff151da4aebefbb4b03edc9ac18a11158116f76425a2078c0c6f4c1d4174"}
Oct 30 19:32:11.972790 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:11.972364    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ae425f-79c9ff7db7-z7nf7"
Oct 30 19:32:11.972790 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:11.972390    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ae425f-79c9ff7db7-z7nf7" event={"ID":"7753bd1c-f916-40db-a923-eb46e92a548e","Type":"ContainerStarted","Data":"d6de27d69c4218f657a24814d652c79dd780e72cca15159ef63314e07464756a"}
Oct 30 19:32:11.974239 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:11.974204    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ae425f-5885c86dd4-4wvh9" event={"ID":"469e981b-1193-4d89-b98a-bc6f794cb532","Type":"ContainerStarted","Data":"92823e6406886c547be12b046c1d35705116dc334827abd330b1831e7563d89c"}
Oct 30 19:32:11.974374 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:11.974262    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ae425f-5885c86dd4-4wvh9" event={"ID":"469e981b-1193-4d89-b98a-bc6f794cb532","Type":"ContainerStarted","Data":"7bd85e8380caa5a3b3e981fe3eddd8a500b32f9d1512f84e6cc6cab8e4ad97d2"}
Oct 30 19:32:11.974374 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:11.974290    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ae425f-5885c86dd4-4wvh9"
Oct 30 19:32:11.975984 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:11.975953    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5cb1c5cd-5tq7t"
Oct 30 19:32:11.976069 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:11.975977    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5cb1c5cd-5tq7t" event={"ID":"fdd1e39c-3a49-4153-ab0f-e61d1ce33839","Type":"ContainerDied","Data":"2df9a41af23828ba90386638b59d94e4a39fc1911dbb86df3d61f25f01c4429b"}
Oct 30 19:32:11.976069 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:11.976019    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2df9a41af23828ba90386638b59d94e4a39fc1911dbb86df3d61f25f01c4429b"
Oct 30 19:32:12.011478 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:12.011412    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ae425f-79c9ff7db7-z7nf7" podStartSLOduration=2.011393741 podStartE2EDuration="2.011393741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:12.003657517 +0000 UTC m=+951.514717485" watchObservedRunningTime="2026-10-30 19:32:12.011393741 +0000 UTC m=+951.522453708"
Oct 30 19:32:12.053994 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:12.053938    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ae425f-5885c86dd4-4wvh9" podStartSLOduration=2.053919124 podStartE2EDuration="2.053919124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:12.047841853 +0000 UTC m=+951.558901817" watchObservedRunningTime="2026-10-30 19:32:12.053919124 +0000 UTC m=+951.564979095"
Oct 30 19:32:15.206766 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.201914    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6"]
Oct 30 19:32:15.263160 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.249615    5043 cpu_manager.go:414] "containerMap: removing container" podUID="fdd1e39c-3a49-4153-ab0f-e61d1ce33839" containerName="revoke-keys"
Oct 30 19:32:15.263160 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.249648    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdd1e39c-3a49-4153-ab0f-e61d1ce33839" containerName="revoke-keys"
Oct 30 19:32:15.263160 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.250067    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdd1e39c-3a49-4153-ab0f-e61d1ce33839" containerName="revoke-keys"
Oct 30 19:32:15.291326 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.291293    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6"
Oct 30 19:32:15.342319 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.342288    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6"]
Oct 30 19:32:15.352443 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.352407    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7"]
Oct 30 19:32:15.362873 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.362149    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7"
Oct 30 19:32:15.365336 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.365307    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7"]
Oct 30 19:32:15.399710 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.399675    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qj2g6\" (UID: \"9f2b4d7b-0014-49da-b5c2-0cf1944e8796\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6"
Oct 30 19:32:15.399928 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.399719    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qj2g6\" (UID: \"9f2b4d7b-0014-49da-b5c2-0cf1944e8796\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6"
Oct 30 19:32:15.399928 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.399788    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qj2g6\" (UID: \"9f2b4d7b-0014-49da-b5c2-0cf1944e8796\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6"
Oct 30 19:32:15.400043 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.399949    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qj2g6\" (UID: \"9f2b4d7b-0014-49da-b5c2-0cf1944e8796\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6"
Oct 30 19:32:15.400106 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.400083    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qj2g6\" (UID: \"9f2b4d7b-0014-49da-b5c2-0cf1944e8796\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6"
Oct 30 19:32:15.400154 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.400136    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qj2g6\" (UID: \"9f2b4d7b-0014-49da-b5c2-0cf1944e8796\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6"
Oct 30 19:32:15.400238 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.400223    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qj2g6\" (UID: \"9f2b4d7b-0014-49da-b5c2-0cf1944e8796\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6"
Oct 30 19:32:15.400299 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.400253    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvsn4\" (UniqueName: \"kubernetes.io/projected/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-kube-api-access-xvsn4\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qj2g6\" (UID: \"9f2b4d7b-0014-49da-b5c2-0cf1944e8796\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6"
Oct 30 19:32:15.400354 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.400332    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qj2g6\" (UID: \"9f2b4d7b-0014-49da-b5c2-0cf1944e8796\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6"
Oct 30 19:32:15.502116 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.502071    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91978947-9c53-4a67-8f97-36ce17619c17-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5qx7\" (UID: \"91978947-9c53-4a67-8f97-36ce17619c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7"
Oct 30 19:32:15.502314 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.502143    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91978947-9c53-4a67-8f97-36ce17619c17-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5qx7\" (UID: \"91978947-9c53-4a67-8f97-36ce17619c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7"
Oct 30 19:32:15.502314 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.502206    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91978947-9c53-4a67-8f97-36ce17619c17-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5qx7\" (UID: \"91978947-9c53-4a67-8f97-36ce17619c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7"
Oct 30 19:32:15.502314 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.502249    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j29w2\" (UniqueName: \"kubernetes.io/projected/91978947-9c53-4a67-8f97-36ce17619c17-kube-api-access-j29w2\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5qx7\" (UID: \"91978947-9c53-4a67-8f97-36ce17619c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7"
Oct 30 19:32:15.502441 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.502381    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qj2g6\" (UID: \"9f2b4d7b-0014-49da-b5c2-0cf1944e8796\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6"
Oct 30 19:32:15.502441 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.502416    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qj2g6\" (UID: \"9f2b4d7b-0014-49da-b5c2-0cf1944e8796\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6"
Oct 30 19:32:15.502536 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.502442    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91978947-9c53-4a67-8f97-36ce17619c17-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5qx7\" (UID: \"91978947-9c53-4a67-8f97-36ce17619c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7"
Oct 30 19:32:15.502536 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.502468    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qj2g6\" (UID: \"9f2b4d7b-0014-49da-b5c2-0cf1944e8796\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6"
Oct 30 19:32:15.502631 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.502541    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91978947-9c53-4a67-8f97-36ce17619c17-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5qx7\" (UID: \"91978947-9c53-4a67-8f97-36ce17619c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7"
Oct 30 19:32:15.502689 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.502626    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qj2g6\" (UID: \"9f2b4d7b-0014-49da-b5c2-0cf1944e8796\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6"
Oct 30 19:32:15.502757 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.502682    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91978947-9c53-4a67-8f97-36ce17619c17-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5qx7\" (UID: \"91978947-9c53-4a67-8f97-36ce17619c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7"
Oct 30 19:32:15.502812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.502796    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qj2g6\" (UID: \"9f2b4d7b-0014-49da-b5c2-0cf1944e8796\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6"
Oct 30 19:32:15.502865 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.502848    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qj2g6\" (UID: \"9f2b4d7b-0014-49da-b5c2-0cf1944e8796\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6"
Oct 30 19:32:15.502865 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.502856    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qj2g6\" (UID: \"9f2b4d7b-0014-49da-b5c2-0cf1944e8796\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6"
Oct 30 19:32:15.502981 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.502905    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qj2g6\" (UID: \"9f2b4d7b-0014-49da-b5c2-0cf1944e8796\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6"
Oct 30 19:32:15.502981 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.502935    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qj2g6\" (UID: \"9f2b4d7b-0014-49da-b5c2-0cf1944e8796\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6"
Oct 30 19:32:15.502981 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.502972    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvsn4\" (UniqueName: \"kubernetes.io/projected/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-kube-api-access-xvsn4\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qj2g6\" (UID: \"9f2b4d7b-0014-49da-b5c2-0cf1944e8796\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6"
Oct 30 19:32:15.503142 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.503017    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91978947-9c53-4a67-8f97-36ce17619c17-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5qx7\" (UID: \"91978947-9c53-4a67-8f97-36ce17619c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7"
Oct 30 19:32:15.503142 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.503057    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91978947-9c53-4a67-8f97-36ce17619c17-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5qx7\" (UID: \"91978947-9c53-4a67-8f97-36ce17619c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7"
Oct 30 19:32:15.503142 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.503098    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qj2g6\" (UID: \"9f2b4d7b-0014-49da-b5c2-0cf1944e8796\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6"
Oct 30 19:32:15.503142 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.503115    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qj2g6\" (UID: \"9f2b4d7b-0014-49da-b5c2-0cf1944e8796\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6"
Oct 30 19:32:15.503595 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.503573    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qj2g6\" (UID: \"9f2b4d7b-0014-49da-b5c2-0cf1944e8796\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6"
Oct 30 19:32:15.503771 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.503710    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qj2g6\" (UID: \"9f2b4d7b-0014-49da-b5c2-0cf1944e8796\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6"
Oct 30 19:32:15.505258 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.505237    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qj2g6\" (UID: \"9f2b4d7b-0014-49da-b5c2-0cf1944e8796\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6"
Oct 30 19:32:15.505484 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.505467    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qj2g6\" (UID: \"9f2b4d7b-0014-49da-b5c2-0cf1944e8796\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6"
Oct 30 19:32:15.512388 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.512359    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qj2g6\" (UID: \"9f2b4d7b-0014-49da-b5c2-0cf1944e8796\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6"
Oct 30 19:32:15.512531 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.512511    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvsn4\" (UniqueName: \"kubernetes.io/projected/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-kube-api-access-xvsn4\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qj2g6\" (UID: \"9f2b4d7b-0014-49da-b5c2-0cf1944e8796\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6"
Oct 30 19:32:15.608253 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.608201    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j29w2\" (UniqueName: \"kubernetes.io/projected/91978947-9c53-4a67-8f97-36ce17619c17-kube-api-access-j29w2\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5qx7\" (UID: \"91978947-9c53-4a67-8f97-36ce17619c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7"
Oct 30 19:32:15.608467 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.608338    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91978947-9c53-4a67-8f97-36ce17619c17-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5qx7\" (UID: \"91978947-9c53-4a67-8f97-36ce17619c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7"
Oct 30 19:32:15.608467 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.608373    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91978947-9c53-4a67-8f97-36ce17619c17-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5qx7\" (UID: \"91978947-9c53-4a67-8f97-36ce17619c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7"
Oct 30 19:32:15.608467 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.608457    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91978947-9c53-4a67-8f97-36ce17619c17-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5qx7\" (UID: \"91978947-9c53-4a67-8f97-36ce17619c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7"
Oct 30 19:32:15.608637 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.608577    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91978947-9c53-4a67-8f97-36ce17619c17-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5qx7\" (UID: \"91978947-9c53-4a67-8f97-36ce17619c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7"
Oct 30 19:32:15.608637 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.608614    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91978947-9c53-4a67-8f97-36ce17619c17-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5qx7\" (UID: \"91978947-9c53-4a67-8f97-36ce17619c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7"
Oct 30 19:32:15.608780 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.608762    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91978947-9c53-4a67-8f97-36ce17619c17-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5qx7\" (UID: \"91978947-9c53-4a67-8f97-36ce17619c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7"
Oct 30 19:32:15.608912 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.608892    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91978947-9c53-4a67-8f97-36ce17619c17-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5qx7\" (UID: \"91978947-9c53-4a67-8f97-36ce17619c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7"
Oct 30 19:32:15.609170 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.609148    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91978947-9c53-4a67-8f97-36ce17619c17-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5qx7\" (UID: \"91978947-9c53-4a67-8f97-36ce17619c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7"
Oct 30 19:32:15.609268 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.609232    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91978947-9c53-4a67-8f97-36ce17619c17-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5qx7\" (UID: \"91978947-9c53-4a67-8f97-36ce17619c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7"
Oct 30 19:32:15.609328 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.609003    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91978947-9c53-4a67-8f97-36ce17619c17-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5qx7\" (UID: \"91978947-9c53-4a67-8f97-36ce17619c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7"
Oct 30 19:32:15.609328 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.609159    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91978947-9c53-4a67-8f97-36ce17619c17-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5qx7\" (UID: \"91978947-9c53-4a67-8f97-36ce17619c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7"
Oct 30 19:32:15.610003 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.609965    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91978947-9c53-4a67-8f97-36ce17619c17-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5qx7\" (UID: \"91978947-9c53-4a67-8f97-36ce17619c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7"
Oct 30 19:32:15.610003 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.609965    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91978947-9c53-4a67-8f97-36ce17619c17-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5qx7\" (UID: \"91978947-9c53-4a67-8f97-36ce17619c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7"
Oct 30 19:32:15.612619 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.612560    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91978947-9c53-4a67-8f97-36ce17619c17-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5qx7\" (UID: \"91978947-9c53-4a67-8f97-36ce17619c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7"
Oct 30 19:32:15.613263 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.613242    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91978947-9c53-4a67-8f97-36ce17619c17-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5qx7\" (UID: \"91978947-9c53-4a67-8f97-36ce17619c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7"
Oct 30 19:32:15.617880 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.617831    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91978947-9c53-4a67-8f97-36ce17619c17-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5qx7\" (UID: \"91978947-9c53-4a67-8f97-36ce17619c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7"
Oct 30 19:32:15.623362 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.623053    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6"
Oct 30 19:32:15.628300 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.628020    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j29w2\" (UniqueName: \"kubernetes.io/projected/91978947-9c53-4a67-8f97-36ce17619c17-kube-api-access-j29w2\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f5qx7\" (UID: \"91978947-9c53-4a67-8f97-36ce17619c17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7"
Oct 30 19:32:15.682338 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.682307    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7"
Oct 30 19:32:15.939133 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:32:15.939088    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f2b4d7b_0014_49da_b5c2_0cf1944e8796.slice/crio-fef565663acbbd414f361b5dde6c4432c087796cef2b19f4faface54bed7fc63 WatchSource:0}: Error finding container fef565663acbbd414f361b5dde6c4432c087796cef2b19f4faface54bed7fc63: Status 404 returned error can't find the container with id fef565663acbbd414f361b5dde6c4432c087796cef2b19f4faface54bed7fc63
Oct 30 19:32:15.944532 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.944487    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:32:15.944654 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.944584    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:32:15.944654 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.944624    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:32:15.945858 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:15.945833    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6"]
Oct 30 19:32:16.021234 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:32:16.021203    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91978947_9c53_4a67_8f97_36ce17619c17.slice/crio-4e9f4945147d160af7faddb12c0dfcb61721070eedba23acb6ffa6b27fa954c1 WatchSource:0}: Error finding container 4e9f4945147d160af7faddb12c0dfcb61721070eedba23acb6ffa6b27fa954c1: Status 404 returned error can't find the container with id 4e9f4945147d160af7faddb12c0dfcb61721070eedba23acb6ffa6b27fa954c1
Oct 30 19:32:16.025560 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:16.025533    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7"]
Oct 30 19:32:16.025833 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:16.025769    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:32:16.025906 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:16.025842    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:32:16.025906 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:16.025880    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:32:16.029882 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:16.029861    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6" event={"ID":"9f2b4d7b-0014-49da-b5c2-0cf1944e8796","Type":"ContainerStarted","Data":"fef565663acbbd414f361b5dde6c4432c087796cef2b19f4faface54bed7fc63"}
Oct 30 19:32:17.166845 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:17.164551    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7" event={"ID":"91978947-9c53-4a67-8f97-36ce17619c17","Type":"ContainerStarted","Data":"808bbc92a0a8928e0fc8262ecaf57243be29af3000a90c546307d58bf27306bd"}
Oct 30 19:32:17.167503 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:17.167474    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7" event={"ID":"91978947-9c53-4a67-8f97-36ce17619c17","Type":"ContainerStarted","Data":"4e9f4945147d160af7faddb12c0dfcb61721070eedba23acb6ffa6b27fa954c1"}
Oct 30 19:32:17.214763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:17.208882    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6" event={"ID":"9f2b4d7b-0014-49da-b5c2-0cf1944e8796","Type":"ContainerStarted","Data":"556e01ffb2066c8435506a921b3660f26e1038f7e9e1ee98fc6bacf6b5401a62"}
Oct 30 19:32:17.228771 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:17.216582    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7" podStartSLOduration=2.216562449 podStartE2EDuration="2.216562449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:17.2142048 +0000 UTC m=+956.725264766" watchObservedRunningTime="2026-10-30 19:32:17.216562449 +0000 UTC m=+956.727622489"
Oct 30 19:32:17.289997 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:17.289922    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6" podStartSLOduration=2.289898822 podStartE2EDuration="2.289898822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:17.242374663 +0000 UTC m=+956.753434633" watchObservedRunningTime="2026-10-30 19:32:17.289898822 +0000 UTC m=+956.800958787"
Oct 30 19:32:17.624187 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:17.624156    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6"
Oct 30 19:32:17.626065 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:17.626045    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qj2g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Oct 30 19:32:17.626217 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:17.626199    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6" podUID="9f2b4d7b-0014-49da-b5c2-0cf1944e8796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Oct 30 19:32:17.691761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:17.691615    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7"
Oct 30 19:32:17.691980 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:17.691781    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-f5qx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 19:32:17.691980 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:17.691829    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7" podUID="91978947-9c53-4a67-8f97-36ce17619c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 19:32:18.626193 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:18.626155    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qj2g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Oct 30 19:32:18.626785 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:18.626729    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6" podUID="9f2b4d7b-0014-49da-b5c2-0cf1944e8796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Oct 30 19:32:18.684203 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:18.684166    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-f5qx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 19:32:18.684487 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:18.684455    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7" podUID="91978947-9c53-4a67-8f97-36ce17619c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 19:32:19.632236 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:19.632201    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qj2g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Oct 30 19:32:19.632884 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:19.632851    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6" podUID="9f2b4d7b-0014-49da-b5c2-0cf1944e8796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Oct 30 19:32:19.684246 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:19.684206    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-f5qx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 19:32:19.684619 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:19.684586    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7" podUID="91978947-9c53-4a67-8f97-36ce17619c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 19:32:19.798724 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:19.790916    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2fd9a937-7c795b4979-8kvvc"
Oct 30 19:32:20.623761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:20.623707    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qj2g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Oct 30 19:32:20.623971 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:20.623804    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6" podUID="9f2b4d7b-0014-49da-b5c2-0cf1944e8796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Oct 30 19:32:20.712760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:20.696833    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-f5qx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 19:32:20.712760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:20.696892    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7" podUID="91978947-9c53-4a67-8f97-36ce17619c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 19:32:21.624296 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:21.624250    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qj2g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Oct 30 19:32:21.624484 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:21.624313    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6" podUID="9f2b4d7b-0014-49da-b5c2-0cf1944e8796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Oct 30 19:32:21.690209 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:21.690167    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-f5qx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 19:32:21.690472 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:21.690445    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7" podUID="91978947-9c53-4a67-8f97-36ce17619c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 19:32:22.623899 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:22.623850    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qj2g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Oct 30 19:32:22.624326 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:22.623914    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6" podUID="9f2b4d7b-0014-49da-b5c2-0cf1944e8796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Oct 30 19:32:22.703764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:22.687187    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-f5qx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 19:32:22.703764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:22.687250    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7" podUID="91978947-9c53-4a67-8f97-36ce17619c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 19:32:23.041788 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:23.041362    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ae425f-5885c86dd4-4wvh9"
Oct 30 19:32:23.053983 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:23.053954    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ae425f-79c9ff7db7-z7nf7"
Oct 30 19:32:23.624354 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:23.624321    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qj2g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Oct 30 19:32:23.624858 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:23.624499    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6" podUID="9f2b4d7b-0014-49da-b5c2-0cf1944e8796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Oct 30 19:32:23.683062 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:23.683026    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-f5qx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 19:32:23.683215 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:23.683092    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7" podUID="91978947-9c53-4a67-8f97-36ce17619c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 19:32:24.624531 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:24.624494    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qj2g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Oct 30 19:32:24.624939 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:24.624560    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6" podUID="9f2b4d7b-0014-49da-b5c2-0cf1944e8796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Oct 30 19:32:24.684028 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:24.683985    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-f5qx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 19:32:24.684201 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:24.684062    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7" podUID="91978947-9c53-4a67-8f97-36ce17619c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 19:32:25.006877 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:32:25.006802    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:32:26.731796 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:26.706859    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qj2g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:32:26.731796 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:26.706927    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6" podUID="9f2b4d7b-0014-49da-b5c2-0cf1944e8796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:27.042094 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:27.041997    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-f5qx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:32:27.066757 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:27.042345    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7" podUID="91978947-9c53-4a67-8f97-36ce17619c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:27.240617 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:27.240463    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7"
Oct 30 19:32:27.241948 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:27.241928    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7"
Oct 30 19:32:27.312819 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:27.291161    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7"
Oct 30 19:32:27.446720 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:27.446687    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6"
Oct 30 19:32:27.494495 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:27.480426    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6"
Oct 30 19:32:27.501679 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:27.500669    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6"
Oct 30 19:32:27.537940 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:27.537904    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5cb1c5cd-5tq7t"]
Oct 30 19:32:27.553701 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:27.553638    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5cb1c5cd-5tq7t"]
Oct 30 19:32:28.648829 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:28.648797    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdd1e39c-3a49-4153-ab0f-e61d1ce33839" path="/var/lib/kubelet/pods/fdd1e39c-3a49-4153-ab0f-e61d1ce33839/volumes"
Oct 30 19:32:29.052292 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:29.052255    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48c6bd74-vptpn"]
Oct 30 19:32:29.063634 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:29.063603    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48c6bd74-vptpn"]
Oct 30 19:32:30.097823 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:30.097788    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-264d6e15-66mzm"]
Oct 30 19:32:30.156980 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:30.156943    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0bd04a4-9d7q2"]
Oct 30 19:32:30.241440 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:30.241404    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bb3c919-fwrpr"]
Oct 30 19:32:30.383818 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:30.383239    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:30.484764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:30.459382    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded"
Oct 30 19:32:30.510761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:30.485665    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:32:30.615761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:30.496318    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-264d6e15-66mzm"]
Oct 30 19:32:30.853462 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:30.853420    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:31.141368 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:31.141258    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:32:31.277799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:31.273957    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:32:31.277799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:31.274494    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:32:31.289121 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:31.289026    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 30 19:32:31.303814 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:31.294434    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:32:31.480816 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:31.417871    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54cdd42b-39c3-4d86-bd89-13e49b1a2aed" path="/var/lib/kubelet/pods/54cdd42b-39c3-4d86-bd89-13e49b1a2aed/volumes"
Oct 30 19:32:31.480816 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:32:31.422596    5043 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-5bb3c919-fwrpr\" is forbidden: User \"system:node:ip-10-0-137-84.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-137-84.us-east-2.compute.internal' and this object" podUID="e766a79e-dff8-4e7e-b6b5-b08f443b8c03" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3c919-fwrpr"
Oct 30 19:32:31.480816 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:31.477084    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee3b76e8-5f8e-4da4-b65c-04abab29e1dc" path="/var/lib/kubelet/pods/ee3b76e8-5f8e-4da4-b65c-04abab29e1dc/volumes"
Oct 30 19:32:31.480816 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:31.477709    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0bd04a4-9d7q2"]
Oct 30 19:32:31.480816 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:31.477791    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bb3c919-fwrpr"]
Oct 30 19:32:31.480816 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:32:31.478259    5043 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-c0bd04a4-9d7q2\" is forbidden: User \"system:node:ip-10-0-137-84.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-137-84.us-east-2.compute.internal' and this object" podUID="9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bd04a4-9d7q2"
Oct 30 19:32:31.511768 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:31.488862    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56482a81-gpcjk"]
Oct 30 19:32:31.511768 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:31.492047    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded"
Oct 30 19:32:31.511768 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:31.492768    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"
Oct 30 19:32:31.617767 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:31.586235    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:31.617767 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:31.587079    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:32:32.088762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:32.083195    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:32:32.100948 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:32.094054    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:32:32.100948 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:32.099149    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:32.221863 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:32.221815    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56482a81-gpcjk"]
Oct 30 19:32:32.228771 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:32.228301    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:32.285772 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:32.275830    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:32.285772 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:32.275955    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:32:32.285772 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:32.276406    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:32:32.285772 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:32.276471    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:32:32.323031 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:32.322990    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:32.357472 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:32.354907    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:32:32.359634 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:32.325992    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:32.360409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:32.326052    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:32.360849 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:32.360832    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:32:32.361027 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:32.361016    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"
Oct 30 19:32:32.512995 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:32.512964    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:32:32.525695 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:32.525666    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:32:32.534007 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:32.531387    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded"
Oct 30 19:32:32.597350 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:32.587383    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:33.071370 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:33.071313    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bb701c3-0f12-4a31-bc50-ee5c96da70c8" path="/var/lib/kubelet/pods/7bb701c3-0f12-4a31-bc50-ee5c96da70c8/volumes"
Oct 30 19:32:33.072504 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:33.072471    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4" path="/var/lib/kubelet/pods/9010ccb8-0b14-4d0c-b1b3-4832a7f6ddb4/volumes"
Oct 30 19:32:33.073674 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:33.073655    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e766a79e-dff8-4e7e-b6b5-b08f443b8c03" path="/var/lib/kubelet/pods/e766a79e-dff8-4e7e-b6b5-b08f443b8c03/volumes"
Oct 30 19:32:33.075677 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:33.075657    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:32:33.092173 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:33.092136    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:32:33.102113 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:33.101899    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:33.114351 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:33.114308    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:32:33.233756 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:33.231227    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:33.236208 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:33.236181    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:32:33.330312 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:33.329350    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:32:33.334376 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:33.334352    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:32:33.352274 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:33.344210    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:32:33.374768 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:33.362818    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:33.374768 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:33.363576    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"
Oct 30 19:32:33.433758 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:33.430354    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:32:33.473797 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:33.473686    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"
Oct 30 19:32:38.036603 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:38.036568    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-921f1fba-wqgxb"]
Oct 30 19:32:38.042841 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:38.042814    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-921f1fba-wqgxb"]
Oct 30 19:32:38.652980 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:38.652939    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58817e5e-c488-42bb-8de7-76df0d2652fa" path="/var/lib/kubelet/pods/58817e5e-c488-42bb-8de7-76df0d2652fa/volumes"
Oct 30 19:32:39.031318 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:39.031245    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4e57ba6-c6bq7"]
Oct 30 19:32:39.039182 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:39.039150    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4e57ba6-c6bq7"]
Oct 30 19:32:40.029050 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:40.029015    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89f15f13-22hc6"]
Oct 30 19:32:40.035499 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:40.035473    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89f15f13-22hc6"]
Oct 30 19:32:40.683945 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:40.683905    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52d38b83-2cbe-4e80-adae-f837d0c9115b" path="/var/lib/kubelet/pods/52d38b83-2cbe-4e80-adae-f837d0c9115b/volumes"
Oct 30 19:32:40.685932 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:40.685897    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="888f7b8b-5ffa-40b6-b428-ae77ee29c855" path="/var/lib/kubelet/pods/888f7b8b-5ffa-40b6-b428-ae77ee29c855/volumes"
Oct 30 19:32:42.035787 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:42.035710    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9565d20-7wl65"]
Oct 30 19:32:42.041976 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:42.041952    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9565d20-7wl65"]
Oct 30 19:32:42.511416 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:42.511378    5043 generic.go:356] "Generic (PLEG): container finished" podID="0615f18f-9ed4-4d64-bc4f-1a57e15286cf" containerID="c583cc7aee0b638dc10d53a5b958f084610eede17043d5993b893fe520295561" exitCode=0
Oct 30 19:32:42.511624 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:42.511453    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6" event={"ID":"0615f18f-9ed4-4d64-bc4f-1a57e15286cf","Type":"ContainerDied","Data":"c583cc7aee0b638dc10d53a5b958f084610eede17043d5993b893fe520295561"}
Oct 30 19:32:42.512008 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:42.511987    5043 scope.go:122] "RemoveContainer" containerID="c583cc7aee0b638dc10d53a5b958f084610eede17043d5993b893fe520295561"
Oct 30 19:32:42.513803 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:42.513765    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:32:42.513895 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:42.513847    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:32:42.513940 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:42.513894    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:32:42.666099 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:42.666065    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e2e5350-a94f-41bb-bea3-8fcd882b2552" path="/var/lib/kubelet/pods/6e2e5350-a94f-41bb-bea3-8fcd882b2552/volumes"
Oct 30 19:32:43.072839 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:43.068876    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9832672-dzt6l"]
Oct 30 19:32:43.089695 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:43.089201    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9832672-dzt6l"]
Oct 30 19:32:43.527351 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:43.527317    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6" event={"ID":"0615f18f-9ed4-4d64-bc4f-1a57e15286cf","Type":"ContainerStarted","Data":"f5dd80eab41e12d10a569be0d78c28bb4ebd90b8ce71880650c50736fc81e279"}
Oct 30 19:32:43.644659 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:43.644629    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6"
Oct 30 19:32:43.645510 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:43.645479    5043 patch_prober.go:29] interesting pod/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 30 19:32:43.645653 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:43.645568    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6" podUID="0615f18f-9ed4-4d64-bc4f-1a57e15286cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 30 19:32:44.035132 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:44.035093    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61156d8c-x9bk8"]
Oct 30 19:32:44.051551 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:44.051488    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61156d8c-x9bk8"]
Oct 30 19:32:44.644885 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:44.644855    5043 patch_prober.go:29] interesting pod/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 30 19:32:44.645288 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:44.644908    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6" podUID="0615f18f-9ed4-4d64-bc4f-1a57e15286cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 30 19:32:44.646827 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:44.646805    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="789d2679-ca00-4aeb-850a-a81e96ee7bb5" path="/var/lib/kubelet/pods/789d2679-ca00-4aeb-850a-a81e96ee7bb5/volumes"
Oct 30 19:32:44.647258 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:44.647245    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5c72961-9b6a-4ae1-a089-03d8f2824672" path="/var/lib/kubelet/pods/b5c72961-9b6a-4ae1-a089-03d8f2824672/volumes"
Oct 30 19:32:45.509758 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:45.509365    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp"]
Oct 30 19:32:45.533754 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:45.525063    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp"]
Oct 30 19:32:45.533754 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:45.525177    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp"
Oct 30 19:32:45.533754 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:45.530926    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-20e66fa4-openshift-default-dockercfg-7rz7c\""
Oct 30 19:32:45.869866 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:45.842295    5043 patch_prober.go:29] interesting pod/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 30 19:32:45.869866 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:45.842405    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6" podUID="0615f18f-9ed4-4d64-bc4f-1a57e15286cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 30 19:32:45.949771 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:45.942098    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfd35ad6-36ee-44ba-8d74-021c61177b3b-istio-data\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp\" (UID: \"dfd35ad6-36ee-44ba-8d74-021c61177b3b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp"
Oct 30 19:32:45.949771 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:45.942254    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcgk2\" (UniqueName: \"kubernetes.io/projected/dfd35ad6-36ee-44ba-8d74-021c61177b3b-kube-api-access-vcgk2\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp\" (UID: \"dfd35ad6-36ee-44ba-8d74-021c61177b3b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp"
Oct 30 19:32:45.949771 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:45.942284    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfd35ad6-36ee-44ba-8d74-021c61177b3b-istiod-ca-cert\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp\" (UID: \"dfd35ad6-36ee-44ba-8d74-021c61177b3b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp"
Oct 30 19:32:45.949771 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:45.942423    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfd35ad6-36ee-44ba-8d74-021c61177b3b-istio-envoy\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp\" (UID: \"dfd35ad6-36ee-44ba-8d74-021c61177b3b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp"
Oct 30 19:32:45.949771 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:45.942460    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfd35ad6-36ee-44ba-8d74-021c61177b3b-istio-token\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp\" (UID: \"dfd35ad6-36ee-44ba-8d74-021c61177b3b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp"
Oct 30 19:32:45.949771 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:45.942658    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd35ad6-36ee-44ba-8d74-021c61177b3b-credential-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp\" (UID: \"dfd35ad6-36ee-44ba-8d74-021c61177b3b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp"
Oct 30 19:32:45.949771 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:45.942794    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfd35ad6-36ee-44ba-8d74-021c61177b3b-istio-podinfo\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp\" (UID: \"dfd35ad6-36ee-44ba-8d74-021c61177b3b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp"
Oct 30 19:32:45.949771 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:45.942836    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfd35ad6-36ee-44ba-8d74-021c61177b3b-workload-certs\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp\" (UID: \"dfd35ad6-36ee-44ba-8d74-021c61177b3b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp"
Oct 30 19:32:45.949771 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:45.942893    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd35ad6-36ee-44ba-8d74-021c61177b3b-workload-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp\" (UID: \"dfd35ad6-36ee-44ba-8d74-021c61177b3b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp"
Oct 30 19:32:46.064640 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:46.052299    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfd35ad6-36ee-44ba-8d74-021c61177b3b-istio-podinfo\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp\" (UID: \"dfd35ad6-36ee-44ba-8d74-021c61177b3b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp"
Oct 30 19:32:46.064640 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:46.052359    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfd35ad6-36ee-44ba-8d74-021c61177b3b-workload-certs\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp\" (UID: \"dfd35ad6-36ee-44ba-8d74-021c61177b3b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp"
Oct 30 19:32:46.064640 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:46.052420    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd35ad6-36ee-44ba-8d74-021c61177b3b-workload-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp\" (UID: \"dfd35ad6-36ee-44ba-8d74-021c61177b3b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp"
Oct 30 19:32:46.064640 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:46.052545    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfd35ad6-36ee-44ba-8d74-021c61177b3b-istio-data\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp\" (UID: \"dfd35ad6-36ee-44ba-8d74-021c61177b3b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp"
Oct 30 19:32:46.064640 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:46.052641    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcgk2\" (UniqueName: \"kubernetes.io/projected/dfd35ad6-36ee-44ba-8d74-021c61177b3b-kube-api-access-vcgk2\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp\" (UID: \"dfd35ad6-36ee-44ba-8d74-021c61177b3b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp"
Oct 30 19:32:46.064640 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:46.052668    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfd35ad6-36ee-44ba-8d74-021c61177b3b-istiod-ca-cert\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp\" (UID: \"dfd35ad6-36ee-44ba-8d74-021c61177b3b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp"
Oct 30 19:32:46.064640 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:46.052807    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfd35ad6-36ee-44ba-8d74-021c61177b3b-istio-envoy\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp\" (UID: \"dfd35ad6-36ee-44ba-8d74-021c61177b3b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp"
Oct 30 19:32:46.064640 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:46.052856    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfd35ad6-36ee-44ba-8d74-021c61177b3b-istio-token\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp\" (UID: \"dfd35ad6-36ee-44ba-8d74-021c61177b3b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp"
Oct 30 19:32:46.064640 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:46.053038    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd35ad6-36ee-44ba-8d74-021c61177b3b-credential-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp\" (UID: \"dfd35ad6-36ee-44ba-8d74-021c61177b3b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp"
Oct 30 19:32:46.064640 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:46.053558    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd35ad6-36ee-44ba-8d74-021c61177b3b-credential-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp\" (UID: \"dfd35ad6-36ee-44ba-8d74-021c61177b3b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp"
Oct 30 19:32:46.064640 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:46.061452    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfd35ad6-36ee-44ba-8d74-021c61177b3b-workload-certs\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp\" (UID: \"dfd35ad6-36ee-44ba-8d74-021c61177b3b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp"
Oct 30 19:32:46.064640 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:46.061701    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd35ad6-36ee-44ba-8d74-021c61177b3b-workload-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp\" (UID: \"dfd35ad6-36ee-44ba-8d74-021c61177b3b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp"
Oct 30 19:32:46.064640 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:46.061951    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfd35ad6-36ee-44ba-8d74-021c61177b3b-istio-data\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp\" (UID: \"dfd35ad6-36ee-44ba-8d74-021c61177b3b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp"
Oct 30 19:32:46.064640 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:46.063308    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfd35ad6-36ee-44ba-8d74-021c61177b3b-istiod-ca-cert\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp\" (UID: \"dfd35ad6-36ee-44ba-8d74-021c61177b3b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp"
Oct 30 19:32:46.102770 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:46.101069    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcgk2\" (UniqueName: \"kubernetes.io/projected/dfd35ad6-36ee-44ba-8d74-021c61177b3b-kube-api-access-vcgk2\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp\" (UID: \"dfd35ad6-36ee-44ba-8d74-021c61177b3b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp"
Oct 30 19:32:46.137262 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:46.123371    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfd35ad6-36ee-44ba-8d74-021c61177b3b-istio-podinfo\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp\" (UID: \"dfd35ad6-36ee-44ba-8d74-021c61177b3b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp"
Oct 30 19:32:46.137262 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:46.133108    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfd35ad6-36ee-44ba-8d74-021c61177b3b-istio-envoy\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp\" (UID: \"dfd35ad6-36ee-44ba-8d74-021c61177b3b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp"
Oct 30 19:32:46.152079 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:46.152048    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfd35ad6-36ee-44ba-8d74-021c61177b3b-istio-token\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp\" (UID: \"dfd35ad6-36ee-44ba-8d74-021c61177b3b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp"
Oct 30 19:32:46.454537 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:46.454079    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp"
Oct 30 19:32:46.645807 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:46.644682    5043 patch_prober.go:29] interesting pod/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 30 19:32:46.645807 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:46.644775    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6" podUID="0615f18f-9ed4-4d64-bc4f-1a57e15286cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 30 19:32:46.787759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:46.759470    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6"
Oct 30 19:32:47.101719 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:47.101163    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp"]
Oct 30 19:32:47.107209 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:47.107172    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:32:47.107299 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:47.107258    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:32:47.107360 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:47.107299    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:32:47.645136 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:47.645105    5043 patch_prober.go:29] interesting pod/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 30 19:32:47.645359 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:47.645339    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6" podUID="0615f18f-9ed4-4d64-bc4f-1a57e15286cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 30 19:32:47.927054 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:47.926968    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp" event={"ID":"dfd35ad6-36ee-44ba-8d74-021c61177b3b","Type":"ContainerStarted","Data":"b1d3d9f4c5634497a9f52cb9018117c2a4b9b81baf656cf5a88532f62f9eafec"}
Oct 30 19:32:47.927196 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:47.927060    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp" event={"ID":"dfd35ad6-36ee-44ba-8d74-021c61177b3b","Type":"ContainerStarted","Data":"ddb6fedb85e41b832d4c8a38e300c51523d63e97c4c6c712d63b3b6e4ff0ec40"}
Oct 30 19:32:47.953501 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:47.953437    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp" podStartSLOduration=2.953423235 podStartE2EDuration="2.953423235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:47.948652392 +0000 UTC m=+987.459712357" watchObservedRunningTime="2026-10-30 19:32:47.953423235 +0000 UTC m=+987.464483200"
Oct 30 19:32:48.454601 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:48.454349    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp"
Oct 30 19:32:48.498762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:48.487410    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp"
Oct 30 19:32:48.644477 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:48.644443    5043 patch_prober.go:29] interesting pod/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 30 19:32:48.644665 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:48.644493    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6" podUID="0615f18f-9ed4-4d64-bc4f-1a57e15286cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 30 19:32:48.937397 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:48.937359    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp"
Oct 30 19:32:48.938321 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:48.938304    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp"
Oct 30 19:32:49.644387 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:49.644352    5043 patch_prober.go:29] interesting pod/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 30 19:32:49.644882 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:49.644413    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6" podUID="0615f18f-9ed4-4d64-bc4f-1a57e15286cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 30 19:32:50.757762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:50.757199    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6"
Oct 30 19:32:50.788308 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:50.788030    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6"
Oct 30 19:32:51.038446 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:51.038277    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84d1285b-drd4t"]
Oct 30 19:32:51.045072 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:51.045046    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84d1285b-drd4t"]
Oct 30 19:32:51.970242 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:51.970209    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-7v4rf"]
Oct 30 19:32:51.980147 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:51.979202    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-7v4rf"
Oct 30 19:32:51.996238 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:51.995582    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-20e66fa4\""
Oct 30 19:32:51.998948 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:51.998923    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-7v4rf"]
Oct 30 19:32:52.017972 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:52.017937    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-j8gh8"]
Oct 30 19:32:52.036637 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:52.036607    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-j8gh8"
Oct 30 19:32:52.052803 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:52.052768    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-j8gh8"]
Oct 30 19:32:52.053202 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:52.053167    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-20e66fa4-dockercfg-sh9xk\""
Oct 30 19:32:52.197338 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:52.164157    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/143d86ee-f169-466e-8232-e4669c5fe647-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-20e66fa4-997965c87-7v4rf\" (UID: \"143d86ee-f169-466e-8232-e4669c5fe647\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-7v4rf"
Oct 30 19:32:52.197338 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:52.164300    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5ccw\" (UniqueName: \"kubernetes.io/projected/98147a81-fd6b-423f-9a9c-2765c284d622-kube-api-access-f5ccw\") pod \"payload-processing-e2e-mt-20e66fa4-54557c4fd4-j8gh8\" (UID: \"98147a81-fd6b-423f-9a9c-2765c284d622\") " pod="openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-j8gh8"
Oct 30 19:32:52.197338 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:52.164363    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98147a81-fd6b-423f-9a9c-2765c284d622-plugins-config-volume\") pod \"payload-processing-e2e-mt-20e66fa4-54557c4fd4-j8gh8\" (UID: \"98147a81-fd6b-423f-9a9c-2765c284d622\") " pod="openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-j8gh8"
Oct 30 19:32:52.197338 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:52.164519    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmmbx\" (UniqueName: \"kubernetes.io/projected/143d86ee-f169-466e-8232-e4669c5fe647-kube-api-access-bmmbx\") pod \"payload-pre-processing-e2e-mt-20e66fa4-997965c87-7v4rf\" (UID: \"143d86ee-f169-466e-8232-e4669c5fe647\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-7v4rf"
Oct 30 19:32:52.313092 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:52.312998    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98147a81-fd6b-423f-9a9c-2765c284d622-plugins-config-volume\") pod \"payload-processing-e2e-mt-20e66fa4-54557c4fd4-j8gh8\" (UID: \"98147a81-fd6b-423f-9a9c-2765c284d622\") " pod="openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-j8gh8"
Oct 30 19:32:52.328766 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:52.313554    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmmbx\" (UniqueName: \"kubernetes.io/projected/143d86ee-f169-466e-8232-e4669c5fe647-kube-api-access-bmmbx\") pod \"payload-pre-processing-e2e-mt-20e66fa4-997965c87-7v4rf\" (UID: \"143d86ee-f169-466e-8232-e4669c5fe647\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-7v4rf"
Oct 30 19:32:52.329350 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:52.329317    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/143d86ee-f169-466e-8232-e4669c5fe647-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-20e66fa4-997965c87-7v4rf\" (UID: \"143d86ee-f169-466e-8232-e4669c5fe647\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-7v4rf"
Oct 30 19:32:52.332038 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:52.329629    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5ccw\" (UniqueName: \"kubernetes.io/projected/98147a81-fd6b-423f-9a9c-2765c284d622-kube-api-access-f5ccw\") pod \"payload-processing-e2e-mt-20e66fa4-54557c4fd4-j8gh8\" (UID: \"98147a81-fd6b-423f-9a9c-2765c284d622\") " pod="openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-j8gh8"
Oct 30 19:32:52.332868 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:52.320867    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98147a81-fd6b-423f-9a9c-2765c284d622-plugins-config-volume\") pod \"payload-processing-e2e-mt-20e66fa4-54557c4fd4-j8gh8\" (UID: \"98147a81-fd6b-423f-9a9c-2765c284d622\") " pod="openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-j8gh8"
Oct 30 19:32:52.334088 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:52.334037    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/143d86ee-f169-466e-8232-e4669c5fe647-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-20e66fa4-997965c87-7v4rf\" (UID: \"143d86ee-f169-466e-8232-e4669c5fe647\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-7v4rf"
Oct 30 19:32:52.444477 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:52.442236    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmmbx\" (UniqueName: \"kubernetes.io/projected/143d86ee-f169-466e-8232-e4669c5fe647-kube-api-access-bmmbx\") pod \"payload-pre-processing-e2e-mt-20e66fa4-997965c87-7v4rf\" (UID: \"143d86ee-f169-466e-8232-e4669c5fe647\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-7v4rf"
Oct 30 19:32:52.506535 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:52.499958    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5ccw\" (UniqueName: \"kubernetes.io/projected/98147a81-fd6b-423f-9a9c-2765c284d622-kube-api-access-f5ccw\") pod \"payload-processing-e2e-mt-20e66fa4-54557c4fd4-j8gh8\" (UID: \"98147a81-fd6b-423f-9a9c-2765c284d622\") " pod="openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-j8gh8"
Oct 30 19:32:52.611510 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:52.611423    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-7v4rf"
Oct 30 19:32:52.653333 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:52.653305    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dec8f742-4e6e-4f2d-819f-a30304039c5b" path="/var/lib/kubelet/pods/dec8f742-4e6e-4f2d-819f-a30304039c5b/volumes"
Oct 30 19:32:52.696453 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:52.695315    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-j8gh8"
Oct 30 19:32:53.282409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:53.282340    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e16f494-qrpqh"]
Oct 30 19:32:53.297473 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:53.297420    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e16f494-qrpqh"]
Oct 30 19:32:53.440121 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:53.439707    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-7v4rf"]
Oct 30 19:32:54.172888 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:54.169598    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-j8gh8"]
Oct 30 19:32:54.523763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:54.486960    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:54.802772 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:54.793879    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:32:55.107832 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:55.092420    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:32:55.108147 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:55.108110    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:55.111471 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:55.111441    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93c44a79-bd6c-4c94-a74f-2cbf6d9240c0" path="/var/lib/kubelet/pods/93c44a79-bd6c-4c94-a74f-2cbf6d9240c0/volumes"
Oct 30 19:32:55.162178 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:55.162143    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-7v4rf" event={"ID":"143d86ee-f169-466e-8232-e4669c5fe647","Type":"ContainerStarted","Data":"97236159d8359348e41f3d70b0d1814e5b5e8ecdc434897c907d67db6ffdbce9"}
Oct 30 19:32:55.162771 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:55.162682    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-7v4rf"
Oct 30 19:32:55.307304 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:55.307228    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a476e910-t5dwj"]
Oct 30 19:32:55.333444 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:55.333363    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a476e910-t5dwj"]
Oct 30 19:32:55.335889 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:55.335836    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-7v4rf" podStartSLOduration=4.335819162 podStartE2EDuration="4.335819162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:55.109658457 +0000 UTC m=+994.620718423" watchObservedRunningTime="2026-10-30 19:32:55.335819162 +0000 UTC m=+994.846879137"
Oct 30 19:32:55.384496 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:55.383074    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:55.450532 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:55.432893    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:32:55.485632 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:55.485589    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded"
Oct 30 19:32:55.492620 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:32:55.492165    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:32:55.502091 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:55.492866    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:55.603987 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:55.603945    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:55.795761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:55.794816    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:55.795761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:55.795346    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:32:56.224457 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:56.171397    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:56.224457 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:56.172347    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:32:56.255265 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:56.254913    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qdh96 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:32:56.255265 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:56.254972    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96" podUID="377f3c78-650d-4680-aa0f-0373eba791eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:56.267771 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:56.266820    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-810f290e-pv6pg"]
Oct 30 19:32:56.401867 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:56.392439    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:56.401867 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:56.392937    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:32:56.493605 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:56.424297    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-810f290e-pv6pg"]
Oct 30 19:32:56.493605 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:56.434096    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:56.493605 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:56.436068    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:32:56.664759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:56.617669    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:32:56.900815 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:56.802878    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:57.049505 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:57.017817    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13248701-d6e5-4fb3-bc91-28a028a9b4b3" path="/var/lib/kubelet/pods/13248701-d6e5-4fb3-bc91-28a028a9b4b3/volumes"
Oct 30 19:32:57.071106 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:57.071069    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f52531a-7cb1-416c-86c2-61db7f8128d9" path="/var/lib/kubelet/pods/3f52531a-7cb1-416c-86c2-61db7f8128d9/volumes"
Oct 30 19:32:57.080536 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:57.080410    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-j8gh8" event={"ID":"98147a81-fd6b-423f-9a9c-2765c284d622","Type":"ContainerStarted","Data":"01c64c2d7ac81f20c0f671f72d848316769780b4a012a838bf751d3a1ae60833"}
Oct 30 19:32:57.081036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:57.081015    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-j8gh8"
Oct 30 19:32:57.097017 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:57.094877    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:32:57.097017 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:57.094917    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-j8gh8" event={"ID":"98147a81-fd6b-423f-9a9c-2765c284d622","Type":"ContainerStarted","Data":"90d9adf405975e44dd1de1e682cf15f0732f0a5bbc2c3dc46de5c62b674f9cdd"}
Oct 30 19:32:57.097017 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:57.094950    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-7v4rf" event={"ID":"143d86ee-f169-466e-8232-e4669c5fe647","Type":"ContainerStarted","Data":"d7ac839008133779d04e18a62e828d0c8017363763622557588e97bdf79d54a4"}
Oct 30 19:32:57.200833 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:57.193340    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:32:57.200833 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:57.193719    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": context deadline exceeded"
Oct 30 19:32:57.353759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:57.336389    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:32:57.374921 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:57.352997    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:57.396400 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:57.395522    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:57.417469 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:57.417399    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-j8gh8" podStartSLOduration=6.417363113 podStartE2EDuration="6.417363113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:56.966713525 +0000 UTC m=+996.477773492" watchObservedRunningTime="2026-10-30 19:32:57.417363113 +0000 UTC m=+996.928423080"
Oct 30 19:32:57.432861 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:57.432817    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:57.438773 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:57.434784    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:57.478270 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:57.478231    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:57.479715 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:57.478535    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:57.611768 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:57.609218    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:57.744820 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:57.742334    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:57.891123 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:57.891076    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:57.893841 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:57.893804    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:57.929996 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:57.929910    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-276b6327-nsqgx"]
Oct 30 19:32:58.207165 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:58.207076    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:32:58.276875 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:58.276829    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-276b6327-nsqgx"]
Oct 30 19:32:58.352792 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:58.342426    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-f5qx7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:32:58.352792 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:58.342490    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7" podUID="91978947-9c53-4a67-8f97-36ce17619c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:58.355187 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:58.354256    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:58.357685 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:58.357663    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:32:58.365871 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:58.365820    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:58.366404 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:58.366383    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:32:58.390838 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:58.390810    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:32:58.469821 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:58.443436    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:58.469821 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:58.444041    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:32:58.474587 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:58.474549    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:58.493142 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:58.485965    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded"
Oct 30 19:32:58.493925 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:58.487577    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 30 19:32:58.494352 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:58.494331    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"
Oct 30 19:32:58.494520 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:58.494507    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"
Oct 30 19:32:58.609205 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:58.609158    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qj2g6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:32:58.609648 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:58.609497    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6" podUID="9f2b4d7b-0014-49da-b5c2-0cf1944e8796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:58.941999 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:58.941894    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:32:58.971761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:58.942268    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:32:59.130834 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:59.112360    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:59.247237 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:59.247190    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:59.359459 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:59.358848    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:32:59.422763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:59.400573    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d84bdb9e-82ea-4a0f-a648-042208f16dac" path="/var/lib/kubelet/pods/d84bdb9e-82ea-4a0f-a648-042208f16dac/volumes"
Oct 30 19:32:59.424019 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:59.419061    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:59.475759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:59.439645    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:59.475759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:59.447816    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:32:59.561819 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:59.541090    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:32:59.563987 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:59.542824    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:59.565037 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:59.542871    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 30 19:32:59.746388 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:59.746346    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:32:59.754392 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:59.754367    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"
Oct 30 19:32:59.839232 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:59.839146    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-7v4rf"
Oct 30 19:32:59.924765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:59.896247    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:32:59.934754 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:59.898806    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:59.941812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:32:59.941788    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"
Oct 30 19:33:00.026768 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:00.023826    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:33:00.049696 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:00.049662    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:33:00.051114 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:00.051089    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:33:00.222773 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:00.221409    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:33:00.222773 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:00.221468    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:33:00.292765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:00.290839    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:33:00.343409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:00.343377    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:33:01.318863 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:01.318817    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr"]
Oct 30 19:33:01.333121 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:01.333091    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr"
Oct 30 19:33:01.340346 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:01.340318    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-976b80-openshift-default-dockercfg-6dkb8\""
Oct 30 19:33:01.347709 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:01.347512    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr"]
Oct 30 19:33:01.536763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:01.505419    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5881801f-8f93-4d5f-b303-2c89af131c12-istio-token\") pod \"e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr\" (UID: \"5881801f-8f93-4d5f-b303-2c89af131c12\") " pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr"
Oct 30 19:33:01.541229 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:01.541190    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5881801f-8f93-4d5f-b303-2c89af131c12-workload-socket\") pod \"e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr\" (UID: \"5881801f-8f93-4d5f-b303-2c89af131c12\") " pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr"
Oct 30 19:33:01.541491 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:01.541463    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5881801f-8f93-4d5f-b303-2c89af131c12-istio-podinfo\") pod \"e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr\" (UID: \"5881801f-8f93-4d5f-b303-2c89af131c12\") " pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr"
Oct 30 19:33:01.541648 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:01.541630    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5881801f-8f93-4d5f-b303-2c89af131c12-istiod-ca-cert\") pod \"e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr\" (UID: \"5881801f-8f93-4d5f-b303-2c89af131c12\") " pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr"
Oct 30 19:33:01.541924 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:01.541901    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvxb5\" (UniqueName: \"kubernetes.io/projected/5881801f-8f93-4d5f-b303-2c89af131c12-kube-api-access-kvxb5\") pod \"e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr\" (UID: \"5881801f-8f93-4d5f-b303-2c89af131c12\") " pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr"
Oct 30 19:33:01.542097 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:01.542078    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5881801f-8f93-4d5f-b303-2c89af131c12-istio-data\") pod \"e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr\" (UID: \"5881801f-8f93-4d5f-b303-2c89af131c12\") " pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr"
Oct 30 19:33:01.578818 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:01.572464    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5881801f-8f93-4d5f-b303-2c89af131c12-credential-socket\") pod \"e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr\" (UID: \"5881801f-8f93-4d5f-b303-2c89af131c12\") " pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr"
Oct 30 19:33:01.578818 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:01.574923    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5881801f-8f93-4d5f-b303-2c89af131c12-workload-certs\") pod \"e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr\" (UID: \"5881801f-8f93-4d5f-b303-2c89af131c12\") " pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr"
Oct 30 19:33:01.578818 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:01.574972    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5881801f-8f93-4d5f-b303-2c89af131c12-istio-envoy\") pod \"e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr\" (UID: \"5881801f-8f93-4d5f-b303-2c89af131c12\") " pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr"
Oct 30 19:33:01.711771 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:01.705557    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5881801f-8f93-4d5f-b303-2c89af131c12-credential-socket\") pod \"e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr\" (UID: \"5881801f-8f93-4d5f-b303-2c89af131c12\") " pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr"
Oct 30 19:33:01.723518 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:01.712471    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5881801f-8f93-4d5f-b303-2c89af131c12-credential-socket\") pod \"e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr\" (UID: \"5881801f-8f93-4d5f-b303-2c89af131c12\") " pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr"
Oct 30 19:33:01.724392 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:01.723794    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5881801f-8f93-4d5f-b303-2c89af131c12-workload-certs\") pod \"e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr\" (UID: \"5881801f-8f93-4d5f-b303-2c89af131c12\") " pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr"
Oct 30 19:33:01.724594 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:01.724578    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5881801f-8f93-4d5f-b303-2c89af131c12-istio-envoy\") pod \"e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr\" (UID: \"5881801f-8f93-4d5f-b303-2c89af131c12\") " pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr"
Oct 30 19:33:01.729000 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:01.725562    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5881801f-8f93-4d5f-b303-2c89af131c12-istio-token\") pod \"e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr\" (UID: \"5881801f-8f93-4d5f-b303-2c89af131c12\") " pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr"
Oct 30 19:33:01.792064 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:01.792026    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5881801f-8f93-4d5f-b303-2c89af131c12-workload-socket\") pod \"e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr\" (UID: \"5881801f-8f93-4d5f-b303-2c89af131c12\") " pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr"
Oct 30 19:33:01.793469 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:01.789187    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5881801f-8f93-4d5f-b303-2c89af131c12-istio-token\") pod \"e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr\" (UID: \"5881801f-8f93-4d5f-b303-2c89af131c12\") " pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr"
Oct 30 19:33:01.793469 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:01.792717    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5881801f-8f93-4d5f-b303-2c89af131c12-workload-socket\") pod \"e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr\" (UID: \"5881801f-8f93-4d5f-b303-2c89af131c12\") " pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr"
Oct 30 19:33:01.793469 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:01.724326    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5881801f-8f93-4d5f-b303-2c89af131c12-workload-certs\") pod \"e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr\" (UID: \"5881801f-8f93-4d5f-b303-2c89af131c12\") " pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr"
Oct 30 19:33:01.793758 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:01.793478    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5881801f-8f93-4d5f-b303-2c89af131c12-istio-podinfo\") pod \"e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr\" (UID: \"5881801f-8f93-4d5f-b303-2c89af131c12\") " pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr"
Oct 30 19:33:01.794621 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:01.794574    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5881801f-8f93-4d5f-b303-2c89af131c12-istiod-ca-cert\") pod \"e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr\" (UID: \"5881801f-8f93-4d5f-b303-2c89af131c12\") " pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr"
Oct 30 19:33:01.812835 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:01.793650    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5881801f-8f93-4d5f-b303-2c89af131c12-istiod-ca-cert\") pod \"e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr\" (UID: \"5881801f-8f93-4d5f-b303-2c89af131c12\") " pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr"
Oct 30 19:33:01.812835 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:01.809668    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvxb5\" (UniqueName: \"kubernetes.io/projected/5881801f-8f93-4d5f-b303-2c89af131c12-kube-api-access-kvxb5\") pod \"e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr\" (UID: \"5881801f-8f93-4d5f-b303-2c89af131c12\") " pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr"
Oct 30 19:33:01.812835 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:01.810196    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5881801f-8f93-4d5f-b303-2c89af131c12-istio-data\") pod \"e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr\" (UID: \"5881801f-8f93-4d5f-b303-2c89af131c12\") " pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr"
Oct 30 19:33:01.812835 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:01.812346    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5881801f-8f93-4d5f-b303-2c89af131c12-istio-data\") pod \"e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr\" (UID: \"5881801f-8f93-4d5f-b303-2c89af131c12\") " pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr"
Oct 30 19:33:01.837698 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:01.836342    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5881801f-8f93-4d5f-b303-2c89af131c12-istio-envoy\") pod \"e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr\" (UID: \"5881801f-8f93-4d5f-b303-2c89af131c12\") " pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr"
Oct 30 19:33:01.837698 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:01.836641    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvxb5\" (UniqueName: \"kubernetes.io/projected/5881801f-8f93-4d5f-b303-2c89af131c12-kube-api-access-kvxb5\") pod \"e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr\" (UID: \"5881801f-8f93-4d5f-b303-2c89af131c12\") " pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr"
Oct 30 19:33:01.839912 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:01.839256    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5881801f-8f93-4d5f-b303-2c89af131c12-istio-podinfo\") pod \"e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr\" (UID: \"5881801f-8f93-4d5f-b303-2c89af131c12\") " pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr"
Oct 30 19:33:01.972761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:01.971168    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr"
Oct 30 19:33:02.353041 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:33:02.353003    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5881801f_8f93_4d5f_b303_2c89af131c12.slice/crio-e06103eaa4c38ce4b0a9e173883b6860fc3e86c78b364d2312c967e33140d7d5 WatchSource:0}: Error finding container e06103eaa4c38ce4b0a9e173883b6860fc3e86c78b364d2312c967e33140d7d5: Status 404 returned error can't find the container with id e06103eaa4c38ce4b0a9e173883b6860fc3e86c78b364d2312c967e33140d7d5
Oct 30 19:33:02.355144 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:02.355113    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr"]
Oct 30 19:33:02.356931 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:02.356861    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:33:02.357075 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:02.356942    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:33:02.357075 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:02.356984    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:33:03.127295 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:03.127261    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr" event={"ID":"5881801f-8f93-4d5f-b303-2c89af131c12","Type":"ContainerStarted","Data":"b4ef42071b9b00e236ae88b2f93b6d461af6502e8c788fa7c3857bcd67b0ddd9"}
Oct 30 19:33:03.127477 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:03.127331    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr" event={"ID":"5881801f-8f93-4d5f-b303-2c89af131c12","Type":"ContainerStarted","Data":"e06103eaa4c38ce4b0a9e173883b6860fc3e86c78b364d2312c967e33140d7d5"}
Oct 30 19:33:03.159612 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:03.159564    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr" podStartSLOduration=3.159549543 podStartE2EDuration="3.159549543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:03.15563186 +0000 UTC m=+1002.666691826" watchObservedRunningTime="2026-10-30 19:33:03.159549543 +0000 UTC m=+1002.670609573"
Oct 30 19:33:03.977760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:03.975831    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr"
Oct 30 19:33:03.983994 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:03.983756    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr"
Oct 30 19:33:04.184900 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:04.184395    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr"
Oct 30 19:33:04.201688 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:04.201658    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr"
Oct 30 19:33:04.542067 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:04.541873    5043 patch_prober.go:29] interesting pod/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:33:04.542067 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:04.541933    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m" podUID="a5a07d80-0678-4d80-bbc6-d17cb0ba35cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:33:05.032135 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:05.032097    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68d0bbe2-qns88"]
Oct 30 19:33:05.040000 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:05.039964    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68d0bbe2-qns88"]
Oct 30 19:33:06.652891 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:06.652854    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4540da8d-6c20-4530-a264-b9b8ee80d890" path="/var/lib/kubelet/pods/4540da8d-6c20-4530-a264-b9b8ee80d890/volumes"
Oct 30 19:33:07.021069 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:07.021033    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-976b80-646cd76644-rlkvm"]
Oct 30 19:33:07.046049 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:07.045965    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-976b80-646cd76644-rlkvm"]
Oct 30 19:33:07.046296 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:07.046165    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-976b80-646cd76644-rlkvm"
Oct 30 19:33:07.063872 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:07.063097    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-976b80\""
Oct 30 19:33:07.064253 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:07.064211    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-976b80-dockercfg-nc6g9\""
Oct 30 19:33:07.155966 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:07.155854    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6vhj\" (UniqueName: \"kubernetes.io/projected/741a0acd-9951-4c23-abf6-1c9c22e15366-kube-api-access-w6vhj\") pod \"payload-processing-e2e-shared-b-w3-976b80-646cd76644-rlkvm\" (UID: \"741a0acd-9951-4c23-abf6-1c9c22e15366\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-976b80-646cd76644-rlkvm"
Oct 30 19:33:07.195762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:07.171035    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/741a0acd-9951-4c23-abf6-1c9c22e15366-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-976b80-646cd76644-rlkvm\" (UID: \"741a0acd-9951-4c23-abf6-1c9c22e15366\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-976b80-646cd76644-rlkvm"
Oct 30 19:33:07.321187 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:07.321091    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/741a0acd-9951-4c23-abf6-1c9c22e15366-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-976b80-646cd76644-rlkvm\" (UID: \"741a0acd-9951-4c23-abf6-1c9c22e15366\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-976b80-646cd76644-rlkvm"
Oct 30 19:33:07.321595 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:07.321570    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6vhj\" (UniqueName: \"kubernetes.io/projected/741a0acd-9951-4c23-abf6-1c9c22e15366-kube-api-access-w6vhj\") pod \"payload-processing-e2e-shared-b-w3-976b80-646cd76644-rlkvm\" (UID: \"741a0acd-9951-4c23-abf6-1c9c22e15366\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-976b80-646cd76644-rlkvm"
Oct 30 19:33:07.323398 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:07.323318    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/741a0acd-9951-4c23-abf6-1c9c22e15366-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-976b80-646cd76644-rlkvm\" (UID: \"741a0acd-9951-4c23-abf6-1c9c22e15366\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-976b80-646cd76644-rlkvm"
Oct 30 19:33:07.467096 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:07.467055    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6vhj\" (UniqueName: \"kubernetes.io/projected/741a0acd-9951-4c23-abf6-1c9c22e15366-kube-api-access-w6vhj\") pod \"payload-processing-e2e-shared-b-w3-976b80-646cd76644-rlkvm\" (UID: \"741a0acd-9951-4c23-abf6-1c9c22e15366\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-976b80-646cd76644-rlkvm"
Oct 30 19:33:07.695678 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:07.695582    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-976b80-646cd76644-rlkvm"
Oct 30 19:33:08.104456 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:08.104411    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-976b80-646cd76644-rlkvm"]
Oct 30 19:33:08.111460 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:33:08.111412    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod741a0acd_9951_4c23_abf6_1c9c22e15366.slice/crio-5f00057a07f5622b31f5a86ae626173186927222c1203ae10415a743cb3f1448 WatchSource:0}: Error finding container 5f00057a07f5622b31f5a86ae626173186927222c1203ae10415a743cb3f1448: Status 404 returned error can't find the container with id 5f00057a07f5622b31f5a86ae626173186927222c1203ae10415a743cb3f1448
Oct 30 19:33:08.467052 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:08.467007    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-976b80-646cd76644-rlkvm" event={"ID":"741a0acd-9951-4c23-abf6-1c9c22e15366","Type":"ContainerStarted","Data":"eac0421f5c972cb7ae74a069dc9cbe4fc272a290e5d4aeab35fcf25eb8ee4762"}
Oct 30 19:33:08.467254 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:08.467088    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-976b80-646cd76644-rlkvm"
Oct 30 19:33:08.467254 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:08.467116    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-976b80-646cd76644-rlkvm" event={"ID":"741a0acd-9951-4c23-abf6-1c9c22e15366","Type":"ContainerStarted","Data":"5f00057a07f5622b31f5a86ae626173186927222c1203ae10415a743cb3f1448"}
Oct 30 19:33:08.486586 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:08.486528    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-976b80-646cd76644-rlkvm" podStartSLOduration=2.486512721 podStartE2EDuration="2.486512721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:08.484471842 +0000 UTC m=+1007.995531807" watchObservedRunningTime="2026-10-30 19:33:08.486512721 +0000 UTC m=+1007.997572686"
Oct 30 19:33:08.774017 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:08.773880    5043 patch_prober.go:29] interesting pod/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:33:08.774017 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:08.773941    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf" podUID="40748d3a-8823-40ee-9425-f5e36a8889d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:33:10.080300 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:09.705125    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-j8gh8"
Oct 30 19:33:10.551827 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:10.551782    5043 generic.go:356] "Generic (PLEG): container finished" podID="a5a07d80-0678-4d80-bbc6-d17cb0ba35cf" containerID="bc800dccc98b72bcbb1f47fb11672198a09ae67496bf6bf35a8766e88235a0f1" exitCode=0
Oct 30 19:33:10.552161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:10.552139    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m" event={"ID":"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf","Type":"ContainerDied","Data":"bc800dccc98b72bcbb1f47fb11672198a09ae67496bf6bf35a8766e88235a0f1"}
Oct 30 19:33:10.552710 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:10.552681    5043 scope.go:122] "RemoveContainer" containerID="bc800dccc98b72bcbb1f47fb11672198a09ae67496bf6bf35a8766e88235a0f1"
Oct 30 19:33:10.555054 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:10.555019    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:33:10.555174 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:10.555111    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:33:10.555174 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:10.555158    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:33:11.032712 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:11.032679    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e33616c-znjxf"]
Oct 30 19:33:11.041236 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:11.041207    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e33616c-znjxf"]
Oct 30 19:33:11.597249 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:11.597216    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m" event={"ID":"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf","Type":"ContainerStarted","Data":"fa6e01ab36eebcbfefe09cbbdf10ad3b6d4c66427d4b6abf0cbe86964415fd85"}
Oct 30 19:33:12.503779 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:12.503730    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m"
Oct 30 19:33:12.503962 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:12.503946    5043 patch_prober.go:29] interesting pod/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 19:33:12.504024 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:12.504002    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m" podUID="a5a07d80-0678-4d80-bbc6-d17cb0ba35cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 19:33:12.647605 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:12.647574    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd13b35a-dad6-42a9-b694-af8475b38edc" path="/var/lib/kubelet/pods/cd13b35a-dad6-42a9-b694-af8475b38edc/volumes"
Oct 30 19:33:13.504599 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:13.504553    5043 patch_prober.go:29] interesting pod/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 19:33:13.504842 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:13.504624    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m" podUID="a5a07d80-0678-4d80-bbc6-d17cb0ba35cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 19:33:14.504078 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:14.504040    5043 patch_prober.go:29] interesting pod/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 19:33:14.504495 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:14.504097    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m" podUID="a5a07d80-0678-4d80-bbc6-d17cb0ba35cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 19:33:15.504755 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:15.504693    5043 patch_prober.go:29] interesting pod/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 19:33:15.505272 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:15.504795    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m" podUID="a5a07d80-0678-4d80-bbc6-d17cb0ba35cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 19:33:15.951559 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:15.951419    5043 generic.go:356] "Generic (PLEG): container finished" podID="40748d3a-8823-40ee-9425-f5e36a8889d6" containerID="90c4e9cd64ebdd562a5dc406f8343f7b669002370aa53934d44370f6fcf30262" exitCode=0
Oct 30 19:33:15.955308 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:15.955045    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf" event={"ID":"40748d3a-8823-40ee-9425-f5e36a8889d6","Type":"ContainerDied","Data":"90c4e9cd64ebdd562a5dc406f8343f7b669002370aa53934d44370f6fcf30262"}
Oct 30 19:33:15.969053 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:15.955659    5043 scope.go:122] "RemoveContainer" containerID="90c4e9cd64ebdd562a5dc406f8343f7b669002370aa53934d44370f6fcf30262"
Oct 30 19:33:16.020962 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:16.020916    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:33:16.021302 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:16.021279    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:33:16.021499 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:16.021480    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:33:16.508536 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:16.508015    5043 patch_prober.go:29] interesting pod/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 19:33:16.508536 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:16.508127    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m" podUID="a5a07d80-0678-4d80-bbc6-d17cb0ba35cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 19:33:17.001661 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:17.001624    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf" event={"ID":"40748d3a-8823-40ee-9425-f5e36a8889d6","Type":"ContainerStarted","Data":"cf42adb3acbf7ae83c1c9c86b507995dbef94847b2237dafaa692a1f4752e02a"}
Oct 30 19:33:17.509110 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:17.509082    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m"
Oct 30 19:33:17.509592 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:17.509575    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m"
Oct 30 19:33:17.510363 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:17.510346    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m"
Oct 30 19:33:17.697276 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:17.697225    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf"
Oct 30 19:33:17.698101 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:17.698068    5043 patch_prober.go:29] interesting pod/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 30 19:33:17.698259 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:17.698129    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf" podUID="40748d3a-8823-40ee-9425-f5e36a8889d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 30 19:33:18.697287 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:18.697253    5043 patch_prober.go:29] interesting pod/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 30 19:33:18.697707 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:18.697316    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf" podUID="40748d3a-8823-40ee-9425-f5e36a8889d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 30 19:33:19.489915 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:19.489882    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-976b80-646cd76644-rlkvm"
Oct 30 19:33:19.698159 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:19.698120    5043 patch_prober.go:29] interesting pod/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 30 19:33:19.698795 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:19.698757    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf" podUID="40748d3a-8823-40ee-9425-f5e36a8889d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 30 19:33:20.696982 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:20.696945    5043 patch_prober.go:29] interesting pod/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 30 19:33:20.697181 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:20.697017    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf" podUID="40748d3a-8823-40ee-9425-f5e36a8889d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 30 19:33:21.697596 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:21.697556    5043 patch_prober.go:29] interesting pod/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 30 19:33:21.698123 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:21.697602    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf" podUID="40748d3a-8823-40ee-9425-f5e36a8889d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 30 19:33:22.134059 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:22.132126    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8034062d-vrdkc"]
Oct 30 19:33:22.149963 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:22.149396    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8034062d-vrdkc"]
Oct 30 19:33:22.648120 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:22.648086    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3ad1c21-2943-40d3-bfd8-b206e6ff488f" path="/var/lib/kubelet/pods/e3ad1c21-2943-40d3-bfd8-b206e6ff488f/volumes"
Oct 30 19:33:22.698037 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:22.697998    5043 patch_prober.go:29] interesting pod/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 30 19:33:22.698432 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:22.698079    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf" podUID="40748d3a-8823-40ee-9425-f5e36a8889d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 30 19:33:22.730516 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:22.730487    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf"
Oct 30 19:33:23.722762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:23.722023    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf"
Oct 30 19:33:23.723834 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:23.723782    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf"
Oct 30 19:33:24.507411 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:24.507081    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc"]
Oct 30 19:33:24.515347 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:24.515323    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc"
Oct 30 19:33:24.521169 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:24.520434    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-59f017-openshift-default-dockercfg-4d6qz\""
Oct 30 19:33:24.541249 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:24.540375    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b34f07ce-6edf-4493-b349-ad7916bb93aa-istio-envoy\") pod \"e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc\" (UID: \"b34f07ce-6edf-4493-b349-ad7916bb93aa\") " pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc"
Oct 30 19:33:24.541249 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:24.540507    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b34f07ce-6edf-4493-b349-ad7916bb93aa-workload-certs\") pod \"e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc\" (UID: \"b34f07ce-6edf-4493-b349-ad7916bb93aa\") " pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc"
Oct 30 19:33:24.541249 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:24.540565    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b34f07ce-6edf-4493-b349-ad7916bb93aa-istio-token\") pod \"e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc\" (UID: \"b34f07ce-6edf-4493-b349-ad7916bb93aa\") " pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc"
Oct 30 19:33:24.541249 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:24.540599    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b34f07ce-6edf-4493-b349-ad7916bb93aa-istio-podinfo\") pod \"e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc\" (UID: \"b34f07ce-6edf-4493-b349-ad7916bb93aa\") " pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc"
Oct 30 19:33:24.541249 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:24.540648    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b34f07ce-6edf-4493-b349-ad7916bb93aa-credential-socket\") pod \"e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc\" (UID: \"b34f07ce-6edf-4493-b349-ad7916bb93aa\") " pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc"
Oct 30 19:33:24.541249 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:24.540701    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b34f07ce-6edf-4493-b349-ad7916bb93aa-istiod-ca-cert\") pod \"e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc\" (UID: \"b34f07ce-6edf-4493-b349-ad7916bb93aa\") " pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc"
Oct 30 19:33:24.541249 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:24.540751    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b34f07ce-6edf-4493-b349-ad7916bb93aa-workload-socket\") pod \"e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc\" (UID: \"b34f07ce-6edf-4493-b349-ad7916bb93aa\") " pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc"
Oct 30 19:33:24.541249 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:24.540794    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q2gw\" (UniqueName: \"kubernetes.io/projected/b34f07ce-6edf-4493-b349-ad7916bb93aa-kube-api-access-2q2gw\") pod \"e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc\" (UID: \"b34f07ce-6edf-4493-b349-ad7916bb93aa\") " pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc"
Oct 30 19:33:24.541249 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:24.540854    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b34f07ce-6edf-4493-b349-ad7916bb93aa-istio-data\") pod \"e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc\" (UID: \"b34f07ce-6edf-4493-b349-ad7916bb93aa\") " pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc"
Oct 30 19:33:24.609648 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:24.609614    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc"]
Oct 30 19:33:24.643100 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:24.643064    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b34f07ce-6edf-4493-b349-ad7916bb93aa-workload-certs\") pod \"e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc\" (UID: \"b34f07ce-6edf-4493-b349-ad7916bb93aa\") " pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc"
Oct 30 19:33:24.643349 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:24.643126    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b34f07ce-6edf-4493-b349-ad7916bb93aa-istio-token\") pod \"e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc\" (UID: \"b34f07ce-6edf-4493-b349-ad7916bb93aa\") " pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc"
Oct 30 19:33:24.643349 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:24.643158    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b34f07ce-6edf-4493-b349-ad7916bb93aa-istio-podinfo\") pod \"e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc\" (UID: \"b34f07ce-6edf-4493-b349-ad7916bb93aa\") " pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc"
Oct 30 19:33:24.643349 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:24.643211    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b34f07ce-6edf-4493-b349-ad7916bb93aa-credential-socket\") pod \"e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc\" (UID: \"b34f07ce-6edf-4493-b349-ad7916bb93aa\") " pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc"
Oct 30 19:33:24.643349 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:24.643267    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b34f07ce-6edf-4493-b349-ad7916bb93aa-istiod-ca-cert\") pod \"e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc\" (UID: \"b34f07ce-6edf-4493-b349-ad7916bb93aa\") " pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc"
Oct 30 19:33:24.643349 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:24.643301    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b34f07ce-6edf-4493-b349-ad7916bb93aa-workload-socket\") pod \"e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc\" (UID: \"b34f07ce-6edf-4493-b349-ad7916bb93aa\") " pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc"
Oct 30 19:33:24.643349 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:24.643344    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q2gw\" (UniqueName: \"kubernetes.io/projected/b34f07ce-6edf-4493-b349-ad7916bb93aa-kube-api-access-2q2gw\") pod \"e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc\" (UID: \"b34f07ce-6edf-4493-b349-ad7916bb93aa\") " pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc"
Oct 30 19:33:24.643819 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:24.643401    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b34f07ce-6edf-4493-b349-ad7916bb93aa-istio-data\") pod \"e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc\" (UID: \"b34f07ce-6edf-4493-b349-ad7916bb93aa\") " pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc"
Oct 30 19:33:24.643819 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:24.643666    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b34f07ce-6edf-4493-b349-ad7916bb93aa-istio-envoy\") pod \"e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc\" (UID: \"b34f07ce-6edf-4493-b349-ad7916bb93aa\") " pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc"
Oct 30 19:33:24.646712 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:24.646683    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b34f07ce-6edf-4493-b349-ad7916bb93aa-istiod-ca-cert\") pod \"e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc\" (UID: \"b34f07ce-6edf-4493-b349-ad7916bb93aa\") " pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc"
Oct 30 19:33:24.647158 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:24.647034    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b34f07ce-6edf-4493-b349-ad7916bb93aa-credential-socket\") pod \"e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc\" (UID: \"b34f07ce-6edf-4493-b349-ad7916bb93aa\") " pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc"
Oct 30 19:33:24.647158 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:24.647111    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b34f07ce-6edf-4493-b349-ad7916bb93aa-workload-socket\") pod \"e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc\" (UID: \"b34f07ce-6edf-4493-b349-ad7916bb93aa\") " pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc"
Oct 30 19:33:24.647315 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:24.647177    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b34f07ce-6edf-4493-b349-ad7916bb93aa-istio-data\") pod \"e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc\" (UID: \"b34f07ce-6edf-4493-b349-ad7916bb93aa\") " pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc"
Oct 30 19:33:24.647564 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:24.647525    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b34f07ce-6edf-4493-b349-ad7916bb93aa-workload-certs\") pod \"e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc\" (UID: \"b34f07ce-6edf-4493-b349-ad7916bb93aa\") " pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc"
Oct 30 19:33:24.737856 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:24.649866    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b34f07ce-6edf-4493-b349-ad7916bb93aa-istio-envoy\") pod \"e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc\" (UID: \"b34f07ce-6edf-4493-b349-ad7916bb93aa\") " pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc"
Oct 30 19:33:24.737856 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:24.714388    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b34f07ce-6edf-4493-b349-ad7916bb93aa-istio-podinfo\") pod \"e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc\" (UID: \"b34f07ce-6edf-4493-b349-ad7916bb93aa\") " pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc"
Oct 30 19:33:25.124145 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:25.124048    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q2gw\" (UniqueName: \"kubernetes.io/projected/b34f07ce-6edf-4493-b349-ad7916bb93aa-kube-api-access-2q2gw\") pod \"e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc\" (UID: \"b34f07ce-6edf-4493-b349-ad7916bb93aa\") " pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc"
Oct 30 19:33:25.126906 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:25.126846    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b34f07ce-6edf-4493-b349-ad7916bb93aa-istio-token\") pod \"e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc\" (UID: \"b34f07ce-6edf-4493-b349-ad7916bb93aa\") " pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc"
Oct 30 19:33:25.134677 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:25.134651    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc"
Oct 30 19:33:25.151357 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:33:25.151319    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:33:25.498869 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:33:25.498832    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb34f07ce_6edf_4493_b349_ad7916bb93aa.slice/crio-969cf7e7b03153a1e88d9f0cd4ce722fbee8d70be4c70e26d82b1b7710e9778c WatchSource:0}: Error finding container 969cf7e7b03153a1e88d9f0cd4ce722fbee8d70be4c70e26d82b1b7710e9778c: Status 404 returned error can't find the container with id 969cf7e7b03153a1e88d9f0cd4ce722fbee8d70be4c70e26d82b1b7710e9778c
Oct 30 19:33:25.499291 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:25.499266    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc"]
Oct 30 19:33:25.501880 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:25.501851    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:33:25.501977 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:25.501914    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:33:25.501977 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:25.501944    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:33:26.260060 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:26.259907    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc" event={"ID":"b34f07ce-6edf-4493-b349-ad7916bb93aa","Type":"ContainerStarted","Data":"35d3bb6a619b8fd1b7a20cd5648be3a3d97cbbd1ee8170fd1ebc766131c7f752"}
Oct 30 19:33:26.260060 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:26.260005    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc" event={"ID":"b34f07ce-6edf-4493-b349-ad7916bb93aa","Type":"ContainerStarted","Data":"969cf7e7b03153a1e88d9f0cd4ce722fbee8d70be4c70e26d82b1b7710e9778c"}
Oct 30 19:33:26.296452 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:26.296389    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc" podStartSLOduration=2.296369275 podStartE2EDuration="2.296369275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:26.290315993 +0000 UTC m=+1025.801375958" watchObservedRunningTime="2026-10-30 19:33:26.296369275 +0000 UTC m=+1025.807429240"
Oct 30 19:33:27.135223 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:27.135193    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc"
Oct 30 19:33:27.141521 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:27.141495    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc"
Oct 30 19:33:27.278374 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:27.276775    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc"
Oct 30 19:33:27.278916 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:27.278412    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc"
Oct 30 19:33:30.530815 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:30.530046    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-59f017-694c87888d-lw9vk"]
Oct 30 19:33:30.536255 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:30.536229    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-59f017-694c87888d-lw9vk"
Oct 30 19:33:30.543555 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:30.543521    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-59f017\""
Oct 30 19:33:30.544188 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:30.544166    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-59f017-dockercfg-78hxs\""
Oct 30 19:33:30.549661 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:30.549639    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-59f017-694c87888d-lw9vk"]
Oct 30 19:33:30.606689 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:30.606654    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f47551b-3e96-4cde-a128-757ef1095525-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-59f017-694c87888d-lw9vk\" (UID: \"2f47551b-3e96-4cde-a128-757ef1095525\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-59f017-694c87888d-lw9vk"
Oct 30 19:33:30.607032 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:30.606857    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmdxv\" (UniqueName: \"kubernetes.io/projected/2f47551b-3e96-4cde-a128-757ef1095525-kube-api-access-bmdxv\") pod \"payload-processing-e2e-shared-a-w2-59f017-694c87888d-lw9vk\" (UID: \"2f47551b-3e96-4cde-a128-757ef1095525\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-59f017-694c87888d-lw9vk"
Oct 30 19:33:30.741764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:30.730690    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmdxv\" (UniqueName: \"kubernetes.io/projected/2f47551b-3e96-4cde-a128-757ef1095525-kube-api-access-bmdxv\") pod \"payload-processing-e2e-shared-a-w2-59f017-694c87888d-lw9vk\" (UID: \"2f47551b-3e96-4cde-a128-757ef1095525\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-59f017-694c87888d-lw9vk"
Oct 30 19:33:30.748171 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:30.748132    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f47551b-3e96-4cde-a128-757ef1095525-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-59f017-694c87888d-lw9vk\" (UID: \"2f47551b-3e96-4cde-a128-757ef1095525\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-59f017-694c87888d-lw9vk"
Oct 30 19:33:30.757758 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:30.749459    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f47551b-3e96-4cde-a128-757ef1095525-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-59f017-694c87888d-lw9vk\" (UID: \"2f47551b-3e96-4cde-a128-757ef1095525\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-59f017-694c87888d-lw9vk"
Oct 30 19:33:31.140267 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:31.140232    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmdxv\" (UniqueName: \"kubernetes.io/projected/2f47551b-3e96-4cde-a128-757ef1095525-kube-api-access-bmdxv\") pod \"payload-processing-e2e-shared-a-w2-59f017-694c87888d-lw9vk\" (UID: \"2f47551b-3e96-4cde-a128-757ef1095525\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-59f017-694c87888d-lw9vk"
Oct 30 19:33:31.153811 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:31.153779    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-768af474-j5578"]
Oct 30 19:33:31.184489 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:31.184437    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-768af474-j5578"]
Oct 30 19:33:31.210401 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:31.210369    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-59f017-694c87888d-lw9vk"
Oct 30 19:33:31.853347 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:33:31.853293    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f47551b_3e96_4cde_a128_757ef1095525.slice/crio-850e0d1049895fb9777a1e049972d9cb22f356a47da03c1ad3245c0a061276e9 WatchSource:0}: Error finding container 850e0d1049895fb9777a1e049972d9cb22f356a47da03c1ad3245c0a061276e9: Status 404 returned error can't find the container with id 850e0d1049895fb9777a1e049972d9cb22f356a47da03c1ad3245c0a061276e9
Oct 30 19:33:31.887083 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:31.883761    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-59f017-694c87888d-lw9vk"]
Oct 30 19:33:32.492413 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:32.492375    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-59f017-694c87888d-lw9vk" event={"ID":"2f47551b-3e96-4cde-a128-757ef1095525","Type":"ContainerStarted","Data":"bab150187234c85dbc13601712566c77cd93d9b2ccdb0fd2ea53b0716e71bec9"}
Oct 30 19:33:32.492605 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:32.492442    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-59f017-694c87888d-lw9vk" event={"ID":"2f47551b-3e96-4cde-a128-757ef1095525","Type":"ContainerStarted","Data":"850e0d1049895fb9777a1e049972d9cb22f356a47da03c1ad3245c0a061276e9"}
Oct 30 19:33:32.492605 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:32.492478    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-59f017-694c87888d-lw9vk"
Oct 30 19:33:32.516324 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:32.516269    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-59f017-694c87888d-lw9vk" podStartSLOduration=2.516254786 podStartE2EDuration="2.516254786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:32.512969743 +0000 UTC m=+1032.024029711" watchObservedRunningTime="2026-10-30 19:33:32.516254786 +0000 UTC m=+1032.027314747"
Oct 30 19:33:32.786811 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:32.785949    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f97dffca-f853-4390-8f4a-cbf04856f5e4" path="/var/lib/kubelet/pods/f97dffca-f853-4390-8f4a-cbf04856f5e4/volumes"
Oct 30 19:33:41.603905 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:41.603865    5043 generic.go:356] "Generic (PLEG): container finished" podID="dfd35ad6-36ee-44ba-8d74-021c61177b3b" containerID="b1d3d9f4c5634497a9f52cb9018117c2a4b9b81baf656cf5a88532f62f9eafec" exitCode=0
Oct 30 19:33:41.604470 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:41.603901    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp" event={"ID":"dfd35ad6-36ee-44ba-8d74-021c61177b3b","Type":"ContainerDied","Data":"b1d3d9f4c5634497a9f52cb9018117c2a4b9b81baf656cf5a88532f62f9eafec"}
Oct 30 19:33:41.604542 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:41.604487    5043 scope.go:122] "RemoveContainer" containerID="b1d3d9f4c5634497a9f52cb9018117c2a4b9b81baf656cf5a88532f62f9eafec"
Oct 30 19:33:41.606748 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:41.606704    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:33:41.606908 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:41.606821    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:33:41.606908 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:41.606866    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:33:42.716760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:42.707938    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp" event={"ID":"dfd35ad6-36ee-44ba-8d74-021c61177b3b","Type":"ContainerStarted","Data":"94bfaaa913474ad26a70e390d1e55ba75d2433a5d84c825623a155189b77ad4c"}
Oct 30 19:33:43.454429 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:43.454395    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp"
Oct 30 19:33:43.454615 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:43.454582    5043 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 30 19:33:43.454664 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:43.454627    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp" podUID="dfd35ad6-36ee-44ba-8d74-021c61177b3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 30 19:33:43.507906 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:43.507877    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-59f017-694c87888d-lw9vk"
Oct 30 19:33:44.454984 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:44.454951    5043 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 30 19:33:44.455488 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:44.455019    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp" podUID="dfd35ad6-36ee-44ba-8d74-021c61177b3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 30 19:33:45.454945 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:45.454913    5043 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 30 19:33:45.455136 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:45.454967    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp" podUID="dfd35ad6-36ee-44ba-8d74-021c61177b3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 30 19:33:46.455388 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:46.455357    5043 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 30 19:33:46.467708 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:46.455412    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp" podUID="dfd35ad6-36ee-44ba-8d74-021c61177b3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 30 19:33:46.906799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:46.906763    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-673809b4-ps722"]
Oct 30 19:33:46.912146 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:46.912113    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-673809b4-ps722"
Oct 30 19:33:46.920582 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:46.919057    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:33:46.920582 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:46.919150    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:33:46.920582 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:46.919161    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:33:46.920582 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:46.919217    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:33:46.920582 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:46.919849    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-673809b4-ps722"]
Oct 30 19:33:47.040053 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:47.040021    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/024272f8-924a-4157-a1a5-4c1ebc5ec4b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-673809b4-ps722\" (UID: \"024272f8-924a-4157-a1a5-4c1ebc5ec4b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-673809b4-ps722"
Oct 30 19:33:47.040210 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:47.040192    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/024272f8-924a-4157-a1a5-4c1ebc5ec4b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-673809b4-ps722\" (UID: \"024272f8-924a-4157-a1a5-4c1ebc5ec4b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-673809b4-ps722"
Oct 30 19:33:47.040311 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:47.040297    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knwjw\" (UniqueName: \"kubernetes.io/projected/024272f8-924a-4157-a1a5-4c1ebc5ec4b9-kube-api-access-knwjw\") pod \"maas-api-revoke-sub-673809b4-ps722\" (UID: \"024272f8-924a-4157-a1a5-4c1ebc5ec4b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-673809b4-ps722"
Oct 30 19:33:47.142171 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:47.142139    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/024272f8-924a-4157-a1a5-4c1ebc5ec4b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-673809b4-ps722\" (UID: \"024272f8-924a-4157-a1a5-4c1ebc5ec4b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-673809b4-ps722"
Oct 30 19:33:47.142377 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:47.142255    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/024272f8-924a-4157-a1a5-4c1ebc5ec4b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-673809b4-ps722\" (UID: \"024272f8-924a-4157-a1a5-4c1ebc5ec4b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-673809b4-ps722"
Oct 30 19:33:47.142377 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:47.142330    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knwjw\" (UniqueName: \"kubernetes.io/projected/024272f8-924a-4157-a1a5-4c1ebc5ec4b9-kube-api-access-knwjw\") pod \"maas-api-revoke-sub-673809b4-ps722\" (UID: \"024272f8-924a-4157-a1a5-4c1ebc5ec4b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-673809b4-ps722"
Oct 30 19:33:47.142913 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:47.142888    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/024272f8-924a-4157-a1a5-4c1ebc5ec4b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-673809b4-ps722\" (UID: \"024272f8-924a-4157-a1a5-4c1ebc5ec4b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-673809b4-ps722"
Oct 30 19:33:47.144908 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:47.144888    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/024272f8-924a-4157-a1a5-4c1ebc5ec4b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-673809b4-ps722\" (UID: \"024272f8-924a-4157-a1a5-4c1ebc5ec4b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-673809b4-ps722"
Oct 30 19:33:47.153535 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:47.153515    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knwjw\" (UniqueName: \"kubernetes.io/projected/024272f8-924a-4157-a1a5-4c1ebc5ec4b9-kube-api-access-knwjw\") pod \"maas-api-revoke-sub-673809b4-ps722\" (UID: \"024272f8-924a-4157-a1a5-4c1ebc5ec4b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-673809b4-ps722"
Oct 30 19:33:47.238059 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:47.238025    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-673809b4-ps722"
Oct 30 19:33:47.455262 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:47.454856    5043 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 30 19:33:47.455262 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:47.454917    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp" podUID="dfd35ad6-36ee-44ba-8d74-021c61177b3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 30 19:33:47.800915 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:47.800842    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-673809b4-ps722"]
Oct 30 19:33:47.807485 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:33:47.807459    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod024272f8_924a_4157_a1a5_4c1ebc5ec4b9.slice/crio-47e671fd5e4c4e2d1a3a1f9845fa0bd0b83c9a5abf691964428446d6d8ebd4e8 WatchSource:0}: Error finding container 47e671fd5e4c4e2d1a3a1f9845fa0bd0b83c9a5abf691964428446d6d8ebd4e8: Status 404 returned error can't find the container with id 47e671fd5e4c4e2d1a3a1f9845fa0bd0b83c9a5abf691964428446d6d8ebd4e8
Oct 30 19:33:48.455100 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:48.455065    5043 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 30 19:33:48.455273 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:48.455125    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp" podUID="dfd35ad6-36ee-44ba-8d74-021c61177b3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 30 19:33:48.885231 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:48.885184    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-673809b4-ps722" event={"ID":"024272f8-924a-4157-a1a5-4c1ebc5ec4b9","Type":"ContainerStarted","Data":"e712f16412844fb2165ce8b2c0e2ee7af346e773acd17f690f2668e3ebbee758"}
Oct 30 19:33:48.885622 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:48.885250    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-673809b4-ps722" event={"ID":"024272f8-924a-4157-a1a5-4c1ebc5ec4b9","Type":"ContainerStarted","Data":"47e671fd5e4c4e2d1a3a1f9845fa0bd0b83c9a5abf691964428446d6d8ebd4e8"}
Oct 30 19:33:48.938085 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:48.938055    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp"
Oct 30 19:33:49.460223 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:49.460192    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp"
Oct 30 19:33:49.901310 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:49.901211    5043 generic.go:356] "Generic (PLEG): container finished" podID="024272f8-924a-4157-a1a5-4c1ebc5ec4b9" containerID="e712f16412844fb2165ce8b2c0e2ee7af346e773acd17f690f2668e3ebbee758" exitCode=0
Oct 30 19:33:49.901768 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:49.901301    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-673809b4-ps722" event={"ID":"024272f8-924a-4157-a1a5-4c1ebc5ec4b9","Type":"ContainerDied","Data":"e712f16412844fb2165ce8b2c0e2ee7af346e773acd17f690f2668e3ebbee758"}
Oct 30 19:33:49.903851 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:49.903827    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp"
Oct 30 19:33:50.170704 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:50.170629    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-673809b4-ps722"
Oct 30 19:33:50.274792 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:50.274718    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/024272f8-924a-4157-a1a5-4c1ebc5ec4b9-kube-api-access-knwjw\" (UniqueName: \"kubernetes.io/projected/024272f8-924a-4157-a1a5-4c1ebc5ec4b9-kube-api-access-knwjw\") pod \"024272f8-924a-4157-a1a5-4c1ebc5ec4b9\" (UID: \"024272f8-924a-4157-a1a5-4c1ebc5ec4b9\") "
Oct 30 19:33:50.274979 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:50.274935    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/024272f8-924a-4157-a1a5-4c1ebc5ec4b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/024272f8-924a-4157-a1a5-4c1ebc5ec4b9-maas-api-xks-ca\") pod \"024272f8-924a-4157-a1a5-4c1ebc5ec4b9\" (UID: \"024272f8-924a-4157-a1a5-4c1ebc5ec4b9\") "
Oct 30 19:33:50.275221 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:50.275198    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/024272f8-924a-4157-a1a5-4c1ebc5ec4b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/024272f8-924a-4157-a1a5-4c1ebc5ec4b9-maas-api-openshift-ca\") pod \"024272f8-924a-4157-a1a5-4c1ebc5ec4b9\" (UID: \"024272f8-924a-4157-a1a5-4c1ebc5ec4b9\") "
Oct 30 19:33:50.275677 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:50.275647    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/024272f8-924a-4157-a1a5-4c1ebc5ec4b9-maas-api-openshift-ca" pod "024272f8-924a-4157-a1a5-4c1ebc5ec4b9" (UID: "024272f8-924a-4157-a1a5-4c1ebc5ec4b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:33:50.276274 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:50.276249    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/024272f8-924a-4157-a1a5-4c1ebc5ec4b9-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:50.280836 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:50.279597    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/024272f8-924a-4157-a1a5-4c1ebc5ec4b9-kube-api-access-knwjw" pod "024272f8-924a-4157-a1a5-4c1ebc5ec4b9" (UID: "024272f8-924a-4157-a1a5-4c1ebc5ec4b9"). InnerVolumeSpecName "kube-api-access-knwjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:33:50.280836 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:50.279699    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/024272f8-924a-4157-a1a5-4c1ebc5ec4b9-maas-api-xks-ca" pod "024272f8-924a-4157-a1a5-4c1ebc5ec4b9" (UID: "024272f8-924a-4157-a1a5-4c1ebc5ec4b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:33:50.378065 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:50.378026    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/024272f8-924a-4157-a1a5-4c1ebc5ec4b9-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:50.378065 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:50.378063    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knwjw\" (UniqueName: \"kubernetes.io/projected/024272f8-924a-4157-a1a5-4c1ebc5ec4b9-kube-api-access-knwjw\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:50.920899 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:50.920780    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-673809b4-ps722"
Oct 30 19:33:50.922330 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:50.920790    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-673809b4-ps722" event={"ID":"024272f8-924a-4157-a1a5-4c1ebc5ec4b9","Type":"ContainerDied","Data":"47e671fd5e4c4e2d1a3a1f9845fa0bd0b83c9a5abf691964428446d6d8ebd4e8"}
Oct 30 19:33:50.922514 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:33:50.922497    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47e671fd5e4c4e2d1a3a1f9845fa0bd0b83c9a5abf691964428446d6d8ebd4e8"
Oct 30 19:33:55.113452 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:33:55.113406    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:34:00.019923 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:00.016781    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt"]
Oct 30 19:34:00.019923 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:00.019714    5043 cpu_manager.go:414] "containerMap: removing container" podUID="024272f8-924a-4157-a1a5-4c1ebc5ec4b9" containerName="revoke-keys"
Oct 30 19:34:00.022255 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:00.020421    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="024272f8-924a-4157-a1a5-4c1ebc5ec4b9" containerName="revoke-keys"
Oct 30 19:34:00.022255 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:00.021612    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="024272f8-924a-4157-a1a5-4c1ebc5ec4b9" containerName="revoke-keys"
Oct 30 19:34:00.041851 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:00.041806    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt"
Oct 30 19:34:00.073856 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:00.068086    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-ca9d14-openshift-default-dockercfg-47vd2\""
Oct 30 19:34:00.356845 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:00.352798    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt"]
Oct 30 19:34:00.568028 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:00.546283    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cffde88e-6bb5-4c7c-b408-bbf25e17f381-workload-socket\") pod \"e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt\" (UID: \"cffde88e-6bb5-4c7c-b408-bbf25e17f381\") " pod="openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt"
Oct 30 19:34:00.576047 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:00.576008    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cffde88e-6bb5-4c7c-b408-bbf25e17f381-istio-token\") pod \"e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt\" (UID: \"cffde88e-6bb5-4c7c-b408-bbf25e17f381\") " pod="openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt"
Oct 30 19:34:00.609959 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:00.609855    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jxpf\" (UniqueName: \"kubernetes.io/projected/cffde88e-6bb5-4c7c-b408-bbf25e17f381-kube-api-access-7jxpf\") pod \"e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt\" (UID: \"cffde88e-6bb5-4c7c-b408-bbf25e17f381\") " pod="openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt"
Oct 30 19:34:00.617133 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:00.610340    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cffde88e-6bb5-4c7c-b408-bbf25e17f381-istio-podinfo\") pod \"e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt\" (UID: \"cffde88e-6bb5-4c7c-b408-bbf25e17f381\") " pod="openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt"
Oct 30 19:34:00.617627 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:00.617456    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cffde88e-6bb5-4c7c-b408-bbf25e17f381-istio-envoy\") pod \"e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt\" (UID: \"cffde88e-6bb5-4c7c-b408-bbf25e17f381\") " pod="openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt"
Oct 30 19:34:00.618035 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:00.617988    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cffde88e-6bb5-4c7c-b408-bbf25e17f381-credential-socket\") pod \"e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt\" (UID: \"cffde88e-6bb5-4c7c-b408-bbf25e17f381\") " pod="openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt"
Oct 30 19:34:00.618289 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:00.618251    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cffde88e-6bb5-4c7c-b408-bbf25e17f381-istio-data\") pod \"e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt\" (UID: \"cffde88e-6bb5-4c7c-b408-bbf25e17f381\") " pod="openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt"
Oct 30 19:34:00.618472 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:00.618449    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cffde88e-6bb5-4c7c-b408-bbf25e17f381-istiod-ca-cert\") pod \"e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt\" (UID: \"cffde88e-6bb5-4c7c-b408-bbf25e17f381\") " pod="openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt"
Oct 30 19:34:00.627761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:00.618937    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cffde88e-6bb5-4c7c-b408-bbf25e17f381-workload-certs\") pod \"e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt\" (UID: \"cffde88e-6bb5-4c7c-b408-bbf25e17f381\") " pod="openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt"
Oct 30 19:34:00.719211 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:00.719173    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02944da9-v956g"]
Oct 30 19:34:00.719211 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:00.719211    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02944da9-v956g"]
Oct 30 19:34:00.729367 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:00.729334    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cffde88e-6bb5-4c7c-b408-bbf25e17f381-workload-certs\") pod \"e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt\" (UID: \"cffde88e-6bb5-4c7c-b408-bbf25e17f381\") " pod="openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt"
Oct 30 19:34:00.729549 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:00.729454    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cffde88e-6bb5-4c7c-b408-bbf25e17f381-workload-socket\") pod \"e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt\" (UID: \"cffde88e-6bb5-4c7c-b408-bbf25e17f381\") " pod="openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt"
Oct 30 19:34:00.729549 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:00.729536    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cffde88e-6bb5-4c7c-b408-bbf25e17f381-istio-token\") pod \"e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt\" (UID: \"cffde88e-6bb5-4c7c-b408-bbf25e17f381\") " pod="openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt"
Oct 30 19:34:00.729679 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:00.729573    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jxpf\" (UniqueName: \"kubernetes.io/projected/cffde88e-6bb5-4c7c-b408-bbf25e17f381-kube-api-access-7jxpf\") pod \"e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt\" (UID: \"cffde88e-6bb5-4c7c-b408-bbf25e17f381\") " pod="openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt"
Oct 30 19:34:00.729679 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:00.729613    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cffde88e-6bb5-4c7c-b408-bbf25e17f381-istio-podinfo\") pod \"e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt\" (UID: \"cffde88e-6bb5-4c7c-b408-bbf25e17f381\") " pod="openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt"
Oct 30 19:34:00.729679 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:00.729660    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cffde88e-6bb5-4c7c-b408-bbf25e17f381-istio-envoy\") pod \"e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt\" (UID: \"cffde88e-6bb5-4c7c-b408-bbf25e17f381\") " pod="openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt"
Oct 30 19:34:00.729896 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:00.729859    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cffde88e-6bb5-4c7c-b408-bbf25e17f381-credential-socket\") pod \"e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt\" (UID: \"cffde88e-6bb5-4c7c-b408-bbf25e17f381\") " pod="openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt"
Oct 30 19:34:00.729896 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:00.729867    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cffde88e-6bb5-4c7c-b408-bbf25e17f381-workload-socket\") pod \"e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt\" (UID: \"cffde88e-6bb5-4c7c-b408-bbf25e17f381\") " pod="openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt"
Oct 30 19:34:00.729896 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:00.729889    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cffde88e-6bb5-4c7c-b408-bbf25e17f381-istio-data\") pod \"e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt\" (UID: \"cffde88e-6bb5-4c7c-b408-bbf25e17f381\") " pod="openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt"
Oct 30 19:34:00.730060 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:00.729908    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cffde88e-6bb5-4c7c-b408-bbf25e17f381-istiod-ca-cert\") pod \"e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt\" (UID: \"cffde88e-6bb5-4c7c-b408-bbf25e17f381\") " pod="openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt"
Oct 30 19:34:00.730114 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:00.730092    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cffde88e-6bb5-4c7c-b408-bbf25e17f381-credential-socket\") pod \"e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt\" (UID: \"cffde88e-6bb5-4c7c-b408-bbf25e17f381\") " pod="openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt"
Oct 30 19:34:00.730208 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:00.730189    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cffde88e-6bb5-4c7c-b408-bbf25e17f381-workload-certs\") pod \"e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt\" (UID: \"cffde88e-6bb5-4c7c-b408-bbf25e17f381\") " pod="openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt"
Oct 30 19:34:00.730358 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:00.730335    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cffde88e-6bb5-4c7c-b408-bbf25e17f381-istio-data\") pod \"e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt\" (UID: \"cffde88e-6bb5-4c7c-b408-bbf25e17f381\") " pod="openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt"
Oct 30 19:34:00.730670 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:00.730652    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cffde88e-6bb5-4c7c-b408-bbf25e17f381-istiod-ca-cert\") pod \"e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt\" (UID: \"cffde88e-6bb5-4c7c-b408-bbf25e17f381\") " pod="openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt"
Oct 30 19:34:00.732234 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:00.732213    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cffde88e-6bb5-4c7c-b408-bbf25e17f381-istio-envoy\") pod \"e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt\" (UID: \"cffde88e-6bb5-4c7c-b408-bbf25e17f381\") " pod="openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt"
Oct 30 19:34:00.732437 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:00.732418    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cffde88e-6bb5-4c7c-b408-bbf25e17f381-istio-podinfo\") pod \"e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt\" (UID: \"cffde88e-6bb5-4c7c-b408-bbf25e17f381\") " pod="openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt"
Oct 30 19:34:00.738073 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:00.738045    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cffde88e-6bb5-4c7c-b408-bbf25e17f381-istio-token\") pod \"e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt\" (UID: \"cffde88e-6bb5-4c7c-b408-bbf25e17f381\") " pod="openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt"
Oct 30 19:34:00.738350 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:00.738333    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jxpf\" (UniqueName: \"kubernetes.io/projected/cffde88e-6bb5-4c7c-b408-bbf25e17f381-kube-api-access-7jxpf\") pod \"e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt\" (UID: \"cffde88e-6bb5-4c7c-b408-bbf25e17f381\") " pod="openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt"
Oct 30 19:34:00.876421 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:00.876064    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt"
Oct 30 19:34:01.351695 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:34:01.351638    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcffde88e_6bb5_4c7c_b408_bbf25e17f381.slice/crio-1e47a59df30cf885048696da9a6620b682978d514c43a4cd74992656f3a8a1ba WatchSource:0}: Error finding container 1e47a59df30cf885048696da9a6620b682978d514c43a4cd74992656f3a8a1ba: Status 404 returned error can't find the container with id 1e47a59df30cf885048696da9a6620b682978d514c43a4cd74992656f3a8a1ba
Oct 30 19:34:01.352162 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:01.352137    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt"]
Oct 30 19:34:01.355310 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:01.355258    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:34:01.355410 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:01.355366    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:34:01.355449 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:01.355420    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:34:01.542947 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:01.542916    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fd9a937-j8w5c"]
Oct 30 19:34:01.548052 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:01.548021    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fd9a937-j8w5c"
Oct 30 19:34:01.552401 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:01.552375    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:34:01.552541 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:01.552489    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:34:01.552541 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:01.552531    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:34:01.552620 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:01.552595    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:34:01.556803 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:01.556775    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fd9a937-j8w5c"]
Oct 30 19:34:01.657338 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:01.657234    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zksw\" (UniqueName: \"kubernetes.io/projected/38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0-kube-api-access-9zksw\") pod \"maas-api-revoke-keys-e2e-mt-2fd9a937-j8w5c\" (UID: \"38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fd9a937-j8w5c"
Oct 30 19:34:01.657538 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:01.657512    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-2fd9a937-j8w5c\" (UID: \"38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fd9a937-j8w5c"
Oct 30 19:34:01.657666 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:01.657645    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-2fd9a937-j8w5c\" (UID: \"38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fd9a937-j8w5c"
Oct 30 19:34:01.776825 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:01.775216    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt" event={"ID":"cffde88e-6bb5-4c7c-b408-bbf25e17f381","Type":"ContainerStarted","Data":"28d793449ee82dc6b14147dfd838d174b5e0f7c7ab9978608299f0c911c6115e"}
Oct 30 19:34:01.776825 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:01.775290    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt" event={"ID":"cffde88e-6bb5-4c7c-b408-bbf25e17f381","Type":"ContainerStarted","Data":"1e47a59df30cf885048696da9a6620b682978d514c43a4cd74992656f3a8a1ba"}
Oct 30 19:34:01.778666 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:01.778619    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-2fd9a937-j8w5c\" (UID: \"38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fd9a937-j8w5c"
Oct 30 19:34:01.798677 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:01.798641    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-2fd9a937-j8w5c\" (UID: \"38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fd9a937-j8w5c"
Oct 30 19:34:01.799232 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:01.799206    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zksw\" (UniqueName: \"kubernetes.io/projected/38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0-kube-api-access-9zksw\") pod \"maas-api-revoke-keys-e2e-mt-2fd9a937-j8w5c\" (UID: \"38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fd9a937-j8w5c"
Oct 30 19:34:01.799518 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:01.799491    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-2fd9a937-j8w5c\" (UID: \"38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fd9a937-j8w5c"
Oct 30 19:34:01.817991 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:01.804415    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-2fd9a937-j8w5c\" (UID: \"38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fd9a937-j8w5c"
Oct 30 19:34:01.846686 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:01.846620    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt" podStartSLOduration=2.846597762 podStartE2EDuration="2.846597762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:01.845168023 +0000 UTC m=+1061.356227993" watchObservedRunningTime="2026-10-30 19:34:01.846597762 +0000 UTC m=+1061.357657742"
Oct 30 19:34:01.853646 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:01.853592    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zksw\" (UniqueName: \"kubernetes.io/projected/38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0-kube-api-access-9zksw\") pod \"maas-api-revoke-keys-e2e-mt-2fd9a937-j8w5c\" (UID: \"38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fd9a937-j8w5c"
Oct 30 19:34:01.878179 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:01.878141    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt"
Oct 30 19:34:01.879612 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:01.879582    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fd9a937-j8w5c"
Oct 30 19:34:02.351233 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:02.351201    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fd9a937-j8w5c"]
Oct 30 19:34:02.351352 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:34:02.351249    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38eb9929_a6b3_45f6_8fa0_c8a26abf5ee0.slice/crio-7e438000164def7feb2813b05b48c2d947a10ce24e1cc61d4bf123c491955a18 WatchSource:0}: Error finding container 7e438000164def7feb2813b05b48c2d947a10ce24e1cc61d4bf123c491955a18: Status 404 returned error can't find the container with id 7e438000164def7feb2813b05b48c2d947a10ce24e1cc61d4bf123c491955a18
Oct 30 19:34:02.683889 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:02.682515    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92500b73-8bdb-4bd9-8770-2a1247803adb" path="/var/lib/kubelet/pods/92500b73-8bdb-4bd9-8770-2a1247803adb/volumes"
Oct 30 19:34:02.799135 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:02.799096    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fd9a937-j8w5c" event={"ID":"38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0","Type":"ContainerStarted","Data":"309c4ffdcd53aed1c8eece2f7d35cc3362b784a2802bcc549b3e38f89f1d6fc3"}
Oct 30 19:34:02.799309 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:02.799166    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fd9a937-j8w5c" event={"ID":"38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0","Type":"ContainerStarted","Data":"7e438000164def7feb2813b05b48c2d947a10ce24e1cc61d4bf123c491955a18"}
Oct 30 19:34:02.818093 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:02.818046    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fd9a937-j8w5c" podStartSLOduration=1.818030538 podStartE2EDuration="1.818030538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:02.817205822 +0000 UTC m=+1062.328265787" watchObservedRunningTime="2026-10-30 19:34:02.818030538 +0000 UTC m=+1062.329090503"
Oct 30 19:34:02.883612 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:02.883580    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt"
Oct 30 19:34:03.814388 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:03.814347    5043 generic.go:356] "Generic (PLEG): container finished" podID="38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0" containerID="309c4ffdcd53aed1c8eece2f7d35cc3362b784a2802bcc549b3e38f89f1d6fc3" exitCode=0
Oct 30 19:34:03.814858 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:03.814393    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fd9a937-j8w5c" event={"ID":"38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0","Type":"ContainerDied","Data":"309c4ffdcd53aed1c8eece2f7d35cc3362b784a2802bcc549b3e38f89f1d6fc3"}
Oct 30 19:34:03.814974 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:03.814954    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt"
Oct 30 19:34:03.816374 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:03.816351    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt"
Oct 30 19:34:04.034592 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:04.034557    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c057c0d-wtvt4"]
Oct 30 19:34:04.044323 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:04.044287    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c057c0d-wtvt4"]
Oct 30 19:34:04.648523 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:04.648487    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6db403a-9d0f-48a8-ad28-7c5b8eb3172e" path="/var/lib/kubelet/pods/e6db403a-9d0f-48a8-ad28-7c5b8eb3172e/volumes"
Oct 30 19:34:05.069409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:05.069365    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fd9a937-j8w5c"
Oct 30 19:34:05.181907 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:05.181818    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0-kube-api-access-9zksw\" (UniqueName: \"kubernetes.io/projected/38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0-kube-api-access-9zksw\") pod \"38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0\" (UID: \"38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0\") "
Oct 30 19:34:05.182088 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:05.181968    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0-maas-api-openshift-ca\") pod \"38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0\" (UID: \"38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0\") "
Oct 30 19:34:05.182088 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:05.182045    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0-maas-api-xks-ca\") pod \"38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0\" (UID: \"38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0\") "
Oct 30 19:34:05.182782 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:05.182722    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0-maas-api-openshift-ca" pod "38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0" (UID: "38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:05.183076 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:05.183024    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:05.185077 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:05.185049    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0-maas-api-xks-ca" pod "38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0" (UID: "38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:34:05.185163 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:05.185114    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0-kube-api-access-9zksw" pod "38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0" (UID: "38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0"). InnerVolumeSpecName "kube-api-access-9zksw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:05.216982 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:05.216942    5043 patch_prober.go:29] interesting pod/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:34:05.217110 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:05.217003    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr" podUID="5881801f-8f93-4d5f-b303-2c89af131c12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:34:05.285536 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:05.285501    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:05.285536 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:05.285539    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zksw\" (UniqueName: \"kubernetes.io/projected/38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0-kube-api-access-9zksw\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:05.596112 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:05.596063    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ca9d14-746f5b5598-6jt2z"]
Oct 30 19:34:05.597441 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:05.597410    5043 cpu_manager.go:414] "containerMap: removing container" podUID="38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0" containerName="revoke-keys"
Oct 30 19:34:05.597441 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:05.597438    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0" containerName="revoke-keys"
Oct 30 19:34:05.597675 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:05.597660    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0" containerName="revoke-keys"
Oct 30 19:34:05.613927 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:05.613794    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca9d14-746f5b5598-6jt2z"
Oct 30 19:34:06.002111 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:06.002078    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-ca9d14\""
Oct 30 19:34:06.007456 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:06.007426    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-ca9d14-dockercfg-l9mq2\""
Oct 30 19:34:06.153030 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:06.152776    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28d04237-1b82-4642-b811-5bd3cd6e2068-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-ca9d14-746f5b5598-6jt2z\" (UID: \"28d04237-1b82-4642-b811-5bd3cd6e2068\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca9d14-746f5b5598-6jt2z"
Oct 30 19:34:06.153547 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:06.153142    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x7rn\" (UniqueName: \"kubernetes.io/projected/28d04237-1b82-4642-b811-5bd3cd6e2068-kube-api-access-8x7rn\") pod \"payload-processing-e2e-shared-b-w2-ca9d14-746f5b5598-6jt2z\" (UID: \"28d04237-1b82-4642-b811-5bd3cd6e2068\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca9d14-746f5b5598-6jt2z"
Oct 30 19:34:06.175774 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:06.175716    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ca9d14-746f5b5598-6jt2z"]
Oct 30 19:34:06.187892 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:06.187850    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca9d14-5c478d7cc5-wtmp4"]
Oct 30 19:34:06.193923 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:06.193891    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fd9a937-j8w5c"
Oct 30 19:34:06.216896 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:06.216861    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca9d14-5c478d7cc5-wtmp4"]
Oct 30 19:34:06.217098 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:06.216910    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fd9a937-j8w5c" event={"ID":"38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0","Type":"ContainerDied","Data":"7e438000164def7feb2813b05b48c2d947a10ce24e1cc61d4bf123c491955a18"}
Oct 30 19:34:06.217098 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:06.216928    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e438000164def7feb2813b05b48c2d947a10ce24e1cc61d4bf123c491955a18"
Oct 30 19:34:06.217098 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:06.216945    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca9d14-5c478d7cc5-wtmp4"
Oct 30 19:34:06.255788 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:06.255665    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x7rn\" (UniqueName: \"kubernetes.io/projected/28d04237-1b82-4642-b811-5bd3cd6e2068-kube-api-access-8x7rn\") pod \"payload-processing-e2e-shared-b-w2-ca9d14-746f5b5598-6jt2z\" (UID: \"28d04237-1b82-4642-b811-5bd3cd6e2068\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca9d14-746f5b5598-6jt2z"
Oct 30 19:34:06.255788 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:06.255715    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9pbn\" (UniqueName: \"kubernetes.io/projected/aaf1c38e-171d-4874-8a38-9f0e03deacce-kube-api-access-c9pbn\") pod \"payload-pre-processing-e2e-shared-b-w2-ca9d14-5c478d7cc5-wtmp4\" (UID: \"aaf1c38e-171d-4874-8a38-9f0e03deacce\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca9d14-5c478d7cc5-wtmp4"
Oct 30 19:34:06.256156 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:06.256112    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaf1c38e-171d-4874-8a38-9f0e03deacce-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-ca9d14-5c478d7cc5-wtmp4\" (UID: \"aaf1c38e-171d-4874-8a38-9f0e03deacce\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca9d14-5c478d7cc5-wtmp4"
Oct 30 19:34:06.256302 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:06.256282    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28d04237-1b82-4642-b811-5bd3cd6e2068-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-ca9d14-746f5b5598-6jt2z\" (UID: \"28d04237-1b82-4642-b811-5bd3cd6e2068\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca9d14-746f5b5598-6jt2z"
Oct 30 19:34:06.257116 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:06.257084    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28d04237-1b82-4642-b811-5bd3cd6e2068-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-ca9d14-746f5b5598-6jt2z\" (UID: \"28d04237-1b82-4642-b811-5bd3cd6e2068\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca9d14-746f5b5598-6jt2z"
Oct 30 19:34:06.275774 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:06.274858    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x7rn\" (UniqueName: \"kubernetes.io/projected/28d04237-1b82-4642-b811-5bd3cd6e2068-kube-api-access-8x7rn\") pod \"payload-processing-e2e-shared-b-w2-ca9d14-746f5b5598-6jt2z\" (UID: \"28d04237-1b82-4642-b811-5bd3cd6e2068\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca9d14-746f5b5598-6jt2z"
Oct 30 19:34:06.358476 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:06.358428    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9pbn\" (UniqueName: \"kubernetes.io/projected/aaf1c38e-171d-4874-8a38-9f0e03deacce-kube-api-access-c9pbn\") pod \"payload-pre-processing-e2e-shared-b-w2-ca9d14-5c478d7cc5-wtmp4\" (UID: \"aaf1c38e-171d-4874-8a38-9f0e03deacce\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca9d14-5c478d7cc5-wtmp4"
Oct 30 19:34:06.358680 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:06.358628    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaf1c38e-171d-4874-8a38-9f0e03deacce-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-ca9d14-5c478d7cc5-wtmp4\" (UID: \"aaf1c38e-171d-4874-8a38-9f0e03deacce\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca9d14-5c478d7cc5-wtmp4"
Oct 30 19:34:06.359689 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:06.359665    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaf1c38e-171d-4874-8a38-9f0e03deacce-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-ca9d14-5c478d7cc5-wtmp4\" (UID: \"aaf1c38e-171d-4874-8a38-9f0e03deacce\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca9d14-5c478d7cc5-wtmp4"
Oct 30 19:34:06.368880 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:06.368847    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9pbn\" (UniqueName: \"kubernetes.io/projected/aaf1c38e-171d-4874-8a38-9f0e03deacce-kube-api-access-c9pbn\") pod \"payload-pre-processing-e2e-shared-b-w2-ca9d14-5c478d7cc5-wtmp4\" (UID: \"aaf1c38e-171d-4874-8a38-9f0e03deacce\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca9d14-5c478d7cc5-wtmp4"
Oct 30 19:34:06.493374 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:06.493324    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca9d14-746f5b5598-6jt2z"
Oct 30 19:34:06.566535 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:06.553977    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca9d14-5c478d7cc5-wtmp4"
Oct 30 19:34:06.859436 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:06.857942    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ca9d14-746f5b5598-6jt2z"]
Oct 30 19:34:06.889952 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:06.889924    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca9d14-5c478d7cc5-wtmp4"]
Oct 30 19:34:07.214407 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:07.214370    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca9d14-5c478d7cc5-wtmp4" event={"ID":"aaf1c38e-171d-4874-8a38-9f0e03deacce","Type":"ContainerStarted","Data":"2b97ea60a086796e4953cabc498fc45cca0017dab4cc37d4370e0b288ab9152f"}
Oct 30 19:34:07.214845 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:07.214429    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca9d14-5c478d7cc5-wtmp4" event={"ID":"aaf1c38e-171d-4874-8a38-9f0e03deacce","Type":"ContainerStarted","Data":"7dc704bd1e62d6a8fb2532b0be20b50f46d193aa3ab55464450d862cd0a25ad0"}
Oct 30 19:34:07.214845 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:07.214459    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca9d14-5c478d7cc5-wtmp4"
Oct 30 19:34:07.216185 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:07.216164    5043 generic.go:356] "Generic (PLEG): container finished" podID="5881801f-8f93-4d5f-b303-2c89af131c12" containerID="b4ef42071b9b00e236ae88b2f93b6d461af6502e8c788fa7c3857bcd67b0ddd9" exitCode=0
Oct 30 19:34:07.216300 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:07.216239    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr" event={"ID":"5881801f-8f93-4d5f-b303-2c89af131c12","Type":"ContainerDied","Data":"b4ef42071b9b00e236ae88b2f93b6d461af6502e8c788fa7c3857bcd67b0ddd9"}
Oct 30 19:34:07.216642 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:07.216625    5043 scope.go:122] "RemoveContainer" containerID="b4ef42071b9b00e236ae88b2f93b6d461af6502e8c788fa7c3857bcd67b0ddd9"
Oct 30 19:34:07.218166 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:07.218142    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca9d14-746f5b5598-6jt2z" event={"ID":"28d04237-1b82-4642-b811-5bd3cd6e2068","Type":"ContainerStarted","Data":"b4bac5d3ae6fad4260ba95633b7a1157f04aa5a2294b863aaaf6216e03b889b5"}
Oct 30 19:34:07.218299 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:07.218182    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca9d14-746f5b5598-6jt2z" event={"ID":"28d04237-1b82-4642-b811-5bd3cd6e2068","Type":"ContainerStarted","Data":"d867b5b6a513f9ce3a48c3081c73239ac96886f215b6a7eaf9a45ec77f32508b"}
Oct 30 19:34:07.218299 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:07.218286    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca9d14-746f5b5598-6jt2z"
Oct 30 19:34:07.218411 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:07.218339    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:34:07.218465 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:07.218451    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:34:07.218522 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:07.218504    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:34:07.262222 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:07.262049    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca9d14-5c478d7cc5-wtmp4" podStartSLOduration=2.262030124 podStartE2EDuration="2.262030124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:07.251917282 +0000 UTC m=+1066.762977247" watchObservedRunningTime="2026-10-30 19:34:07.262030124 +0000 UTC m=+1066.773090090"
Oct 30 19:34:07.284707 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:07.284657    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca9d14-746f5b5598-6jt2z" podStartSLOduration=2.284643625 podStartE2EDuration="2.284643625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:07.277819266 +0000 UTC m=+1066.788879231" watchObservedRunningTime="2026-10-30 19:34:07.284643625 +0000 UTC m=+1066.795703590"
Oct 30 19:34:08.032711 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:08.032677    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01637b35-5sckc"]
Oct 30 19:34:08.038909 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:08.038878    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01637b35-5sckc"]
Oct 30 19:34:08.230978 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:08.230939    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr" event={"ID":"5881801f-8f93-4d5f-b303-2c89af131c12","Type":"ContainerStarted","Data":"6356573d12eb697722f991ee6d9aed500fafd31e7e2f49bc3f8521d6b5be8127"}
Oct 30 19:34:08.707670 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:08.707627    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d" path="/var/lib/kubelet/pods/0e94c7a3-bff9-47ff-b6f1-d907e6c06a4d/volumes"
Oct 30 19:34:08.985761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:08.980258    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr"
Oct 30 19:34:08.985990 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:08.985899    5043 patch_prober.go:29] interesting pod/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 19:34:08.986069 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:08.986012    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr" podUID="5881801f-8f93-4d5f-b303-2c89af131c12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 19:34:09.976191 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:09.976151    5043 patch_prober.go:29] interesting pod/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 19:34:09.976835 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:09.976803    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr" podUID="5881801f-8f93-4d5f-b303-2c89af131c12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 19:34:10.978762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:10.975146    5043 patch_prober.go:29] interesting pod/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 19:34:10.978762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:10.975212    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr" podUID="5881801f-8f93-4d5f-b303-2c89af131c12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 19:34:11.703046 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:11.703011    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2fd9a937-7c795b4979-8kvvc"]
Oct 30 19:34:12.115763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:12.113794    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b9ff6ab-c5t78"]
Oct 30 19:34:12.161648 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:12.151027    5043 patch_prober.go:29] interesting pod/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 19:34:12.161648 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:12.151096    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr" podUID="5881801f-8f93-4d5f-b303-2c89af131c12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 19:34:12.180992 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:12.180956    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b9ff6ab-c5t78"]
Oct 30 19:34:12.696764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:12.696484    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="199728c5-1b99-4baa-b45b-e0470005d82f" path="/var/lib/kubelet/pods/199728c5-1b99-4baa-b45b-e0470005d82f/volumes"
Oct 30 19:34:12.986757 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:12.986180    5043 patch_prober.go:29] interesting pod/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 19:34:12.986757 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:12.986262    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr" podUID="5881801f-8f93-4d5f-b303-2c89af131c12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 19:34:13.975168 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:13.975130    5043 patch_prober.go:29] interesting pod/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 19:34:13.975590 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:13.975216    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr" podUID="5881801f-8f93-4d5f-b303-2c89af131c12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 19:34:14.979535 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:14.979508    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr"
Oct 30 19:34:14.980022 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:14.980002    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr"
Oct 30 19:34:14.980650 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:14.980634    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr"
Oct 30 19:34:15.045802 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:15.045725    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6b64735-c5zv6"]
Oct 30 19:34:15.055264 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:15.055231    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6b64735-c5zv6"]
Oct 30 19:34:16.706880 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:16.706802    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-2fd9a937-7c795b4979-8kvvc" podUID="01e9b019-f11b-4fc9-9044-106429eddcfd" containerName="payload-pre-processing" containerID="cri-o://7722105df0dd6143efe99dfd33200b95b3df1a87432ff15647463ea61535bb1d" gracePeriod=25
Oct 30 19:34:16.791765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:16.788496    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cd8aaed-488b-4236-b372-05d2c3236af1" path="/var/lib/kubelet/pods/5cd8aaed-488b-4236-b372-05d2c3236af1/volumes"
Oct 30 19:34:17.202228 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:17.202195    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2fd9a937-7c795b4979-8kvvc"
Oct 30 19:34:17.214088 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:17.214055    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01e9b019-f11b-4fc9-9044-106429eddcfd-kube-api-access-rwn2x\" (UniqueName: \"kubernetes.io/projected/01e9b019-f11b-4fc9-9044-106429eddcfd-kube-api-access-rwn2x\") pod \"01e9b019-f11b-4fc9-9044-106429eddcfd\" (UID: \"01e9b019-f11b-4fc9-9044-106429eddcfd\") "
Oct 30 19:34:17.214297 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:17.214107    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01e9b019-f11b-4fc9-9044-106429eddcfd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01e9b019-f11b-4fc9-9044-106429eddcfd-plugins-config-volume\") pod \"01e9b019-f11b-4fc9-9044-106429eddcfd\" (UID: \"01e9b019-f11b-4fc9-9044-106429eddcfd\") "
Oct 30 19:34:17.215004 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:17.214972    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01e9b019-f11b-4fc9-9044-106429eddcfd-plugins-config-volume" pod "01e9b019-f11b-4fc9-9044-106429eddcfd" (UID: "01e9b019-f11b-4fc9-9044-106429eddcfd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:17.217585 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:17.217560    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01e9b019-f11b-4fc9-9044-106429eddcfd-kube-api-access-rwn2x" pod "01e9b019-f11b-4fc9-9044-106429eddcfd" (UID: "01e9b019-f11b-4fc9-9044-106429eddcfd"). InnerVolumeSpecName "kube-api-access-rwn2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:17.250811 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:17.250765    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8c67e12-75v4f"]
Oct 30 19:34:17.252124 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:17.252098    5043 cpu_manager.go:414] "containerMap: removing container" podUID="01e9b019-f11b-4fc9-9044-106429eddcfd" containerName="payload-pre-processing"
Oct 30 19:34:17.252124 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:17.252124    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01e9b019-f11b-4fc9-9044-106429eddcfd" containerName="payload-pre-processing"
Oct 30 19:34:17.252317 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:17.252303    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="01e9b019-f11b-4fc9-9044-106429eddcfd" containerName="payload-pre-processing"
Oct 30 19:34:17.259851 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:17.259819    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8c67e12-75v4f"]
Oct 30 19:34:17.260031 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:17.259945    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c67e12-75v4f"
Oct 30 19:34:17.273533 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:17.273503    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:34:17.274670 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:17.274281    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:34:17.274670 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:17.274384    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:34:17.274670 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:17.274470    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:34:17.321339 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:17.319403    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwn2x\" (UniqueName: \"kubernetes.io/projected/01e9b019-f11b-4fc9-9044-106429eddcfd-kube-api-access-rwn2x\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:17.321339 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:17.319441    5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01e9b019-f11b-4fc9-9044-106429eddcfd-plugins-config-volume\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:17.420988 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:17.420951    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98f1f260-d834-4c0f-a2b2-89817b15f1e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8c67e12-75v4f\" (UID: \"98f1f260-d834-4c0f-a2b2-89817b15f1e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c67e12-75v4f"
Oct 30 19:34:17.421192 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:17.421166    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98f1f260-d834-4c0f-a2b2-89817b15f1e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8c67e12-75v4f\" (UID: \"98f1f260-d834-4c0f-a2b2-89817b15f1e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c67e12-75v4f"
Oct 30 19:34:17.421449 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:17.421431    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbmfs\" (UniqueName: \"kubernetes.io/projected/98f1f260-d834-4c0f-a2b2-89817b15f1e9-kube-api-access-sbmfs\") pod \"maas-api-revoke-sub-a8c67e12-75v4f\" (UID: \"98f1f260-d834-4c0f-a2b2-89817b15f1e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c67e12-75v4f"
Oct 30 19:34:17.477973 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:17.477934    5043 generic.go:356] "Generic (PLEG): container finished" podID="01e9b019-f11b-4fc9-9044-106429eddcfd" containerID="7722105df0dd6143efe99dfd33200b95b3df1a87432ff15647463ea61535bb1d" exitCode=0
Oct 30 19:34:17.478172 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:17.477981    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2fd9a937-7c795b4979-8kvvc"
Oct 30 19:34:17.478172 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:17.478022    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2fd9a937-7c795b4979-8kvvc" event={"ID":"01e9b019-f11b-4fc9-9044-106429eddcfd","Type":"ContainerDied","Data":"7722105df0dd6143efe99dfd33200b95b3df1a87432ff15647463ea61535bb1d"}
Oct 30 19:34:17.478172 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:17.478079    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2fd9a937-7c795b4979-8kvvc" event={"ID":"01e9b019-f11b-4fc9-9044-106429eddcfd","Type":"ContainerDied","Data":"3925e0536a2083d7bfd3280808a9657261abd07415fd698e1bfaebdd78dbe621"}
Oct 30 19:34:17.478172 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:17.478108    5043 scope.go:122] "RemoveContainer" containerID="7722105df0dd6143efe99dfd33200b95b3df1a87432ff15647463ea61535bb1d"
Oct 30 19:34:17.491917 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:17.491892    5043 scope.go:122] "RemoveContainer" containerID="7722105df0dd6143efe99dfd33200b95b3df1a87432ff15647463ea61535bb1d"
Oct 30 19:34:17.492257 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:34:17.492235    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7722105df0dd6143efe99dfd33200b95b3df1a87432ff15647463ea61535bb1d\": container with ID starting with 7722105df0dd6143efe99dfd33200b95b3df1a87432ff15647463ea61535bb1d not found: ID does not exist" containerID="7722105df0dd6143efe99dfd33200b95b3df1a87432ff15647463ea61535bb1d"
Oct 30 19:34:17.492345 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:17.492266    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7722105df0dd6143efe99dfd33200b95b3df1a87432ff15647463ea61535bb1d"} err="failed to get container status \"7722105df0dd6143efe99dfd33200b95b3df1a87432ff15647463ea61535bb1d\": rpc error: code = NotFound desc = could not find container \"7722105df0dd6143efe99dfd33200b95b3df1a87432ff15647463ea61535bb1d\": container with ID starting with 7722105df0dd6143efe99dfd33200b95b3df1a87432ff15647463ea61535bb1d not found: ID does not exist"
Oct 30 19:34:17.509326 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:17.509294    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2fd9a937-7c795b4979-8kvvc"]
Oct 30 19:34:17.519048 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:17.519016    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2fd9a937-7c795b4979-8kvvc"]
Oct 30 19:34:17.523088 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:17.523056    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98f1f260-d834-4c0f-a2b2-89817b15f1e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8c67e12-75v4f\" (UID: \"98f1f260-d834-4c0f-a2b2-89817b15f1e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c67e12-75v4f"
Oct 30 19:34:17.523289 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:17.523272    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbmfs\" (UniqueName: \"kubernetes.io/projected/98f1f260-d834-4c0f-a2b2-89817b15f1e9-kube-api-access-sbmfs\") pod \"maas-api-revoke-sub-a8c67e12-75v4f\" (UID: \"98f1f260-d834-4c0f-a2b2-89817b15f1e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c67e12-75v4f"
Oct 30 19:34:17.523397 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:17.523382    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98f1f260-d834-4c0f-a2b2-89817b15f1e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8c67e12-75v4f\" (UID: \"98f1f260-d834-4c0f-a2b2-89817b15f1e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c67e12-75v4f"
Oct 30 19:34:17.524061 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:17.524041    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98f1f260-d834-4c0f-a2b2-89817b15f1e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8c67e12-75v4f\" (UID: \"98f1f260-d834-4c0f-a2b2-89817b15f1e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c67e12-75v4f"
Oct 30 19:34:17.525839 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:17.525817    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98f1f260-d834-4c0f-a2b2-89817b15f1e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8c67e12-75v4f\" (UID: \"98f1f260-d834-4c0f-a2b2-89817b15f1e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c67e12-75v4f"
Oct 30 19:34:17.532590 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:17.532561    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbmfs\" (UniqueName: \"kubernetes.io/projected/98f1f260-d834-4c0f-a2b2-89817b15f1e9-kube-api-access-sbmfs\") pod \"maas-api-revoke-sub-a8c67e12-75v4f\" (UID: \"98f1f260-d834-4c0f-a2b2-89817b15f1e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c67e12-75v4f"
Oct 30 19:34:17.630374 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:17.630275    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c67e12-75v4f"
Oct 30 19:34:17.927655 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:34:17.927624    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98f1f260_d834_4c0f_a2b2_89817b15f1e9.slice/crio-63484a74d484e288f35462adbfb560c0469f4d9f8e1380db6512625d03f8cc38 WatchSource:0}: Error finding container 63484a74d484e288f35462adbfb560c0469f4d9f8e1380db6512625d03f8cc38: Status 404 returned error can't find the container with id 63484a74d484e288f35462adbfb560c0469f4d9f8e1380db6512625d03f8cc38
Oct 30 19:34:17.929041 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:17.929016    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8c67e12-75v4f"]
Oct 30 19:34:18.234930 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:18.234900    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca9d14-5c478d7cc5-wtmp4"
Oct 30 19:34:18.236378 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:18.235959    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca9d14-746f5b5598-6jt2z"
Oct 30 19:34:18.519591 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:18.519369    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c67e12-75v4f" event={"ID":"98f1f260-d834-4c0f-a2b2-89817b15f1e9","Type":"ContainerStarted","Data":"b3a9ccf5a1557f239927d2e47df86d9e2e5ce0d03b5dd22f0164b00fcab8315b"}
Oct 30 19:34:18.519591 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:18.519487    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c67e12-75v4f" event={"ID":"98f1f260-d834-4c0f-a2b2-89817b15f1e9","Type":"ContainerStarted","Data":"63484a74d484e288f35462adbfb560c0469f4d9f8e1380db6512625d03f8cc38"}
Oct 30 19:34:18.541677 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:18.541615    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c67e12-75v4f" podStartSLOduration=1.541598096 podStartE2EDuration="1.541598096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:18.535349209 +0000 UTC m=+1078.046409175" watchObservedRunningTime="2026-10-30 19:34:18.541598096 +0000 UTC m=+1078.052658061"
Oct 30 19:34:18.649309 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:18.649278    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01e9b019-f11b-4fc9-9044-106429eddcfd" path="/var/lib/kubelet/pods/01e9b019-f11b-4fc9-9044-106429eddcfd/volumes"
Oct 30 19:34:19.533424 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:19.533385    5043 generic.go:356] "Generic (PLEG): container finished" podID="98f1f260-d834-4c0f-a2b2-89817b15f1e9" containerID="b3a9ccf5a1557f239927d2e47df86d9e2e5ce0d03b5dd22f0164b00fcab8315b" exitCode=0
Oct 30 19:34:19.533890 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:19.533454    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c67e12-75v4f" event={"ID":"98f1f260-d834-4c0f-a2b2-89817b15f1e9","Type":"ContainerDied","Data":"b3a9ccf5a1557f239927d2e47df86d9e2e5ce0d03b5dd22f0164b00fcab8315b"}
Oct 30 19:34:20.759074 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:20.759050    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c67e12-75v4f"
Oct 30 19:34:20.793490 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:20.793458    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98f1f260-d834-4c0f-a2b2-89817b15f1e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98f1f260-d834-4c0f-a2b2-89817b15f1e9-maas-api-openshift-ca\") pod \"98f1f260-d834-4c0f-a2b2-89817b15f1e9\" (UID: \"98f1f260-d834-4c0f-a2b2-89817b15f1e9\") "
Oct 30 19:34:20.793687 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:20.793570    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98f1f260-d834-4c0f-a2b2-89817b15f1e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98f1f260-d834-4c0f-a2b2-89817b15f1e9-maas-api-xks-ca\") pod \"98f1f260-d834-4c0f-a2b2-89817b15f1e9\" (UID: \"98f1f260-d834-4c0f-a2b2-89817b15f1e9\") "
Oct 30 19:34:20.793860 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:20.793842    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98f1f260-d834-4c0f-a2b2-89817b15f1e9-kube-api-access-sbmfs\" (UniqueName: \"kubernetes.io/projected/98f1f260-d834-4c0f-a2b2-89817b15f1e9-kube-api-access-sbmfs\") pod \"98f1f260-d834-4c0f-a2b2-89817b15f1e9\" (UID: \"98f1f260-d834-4c0f-a2b2-89817b15f1e9\") "
Oct 30 19:34:20.793934 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:20.793883    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98f1f260-d834-4c0f-a2b2-89817b15f1e9-maas-api-openshift-ca" pod "98f1f260-d834-4c0f-a2b2-89817b15f1e9" (UID: "98f1f260-d834-4c0f-a2b2-89817b15f1e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:20.794727 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:20.794661    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98f1f260-d834-4c0f-a2b2-89817b15f1e9-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:20.797189 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:20.797149    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98f1f260-d834-4c0f-a2b2-89817b15f1e9-maas-api-xks-ca" pod "98f1f260-d834-4c0f-a2b2-89817b15f1e9" (UID: "98f1f260-d834-4c0f-a2b2-89817b15f1e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:34:20.797293 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:20.797202    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98f1f260-d834-4c0f-a2b2-89817b15f1e9-kube-api-access-sbmfs" pod "98f1f260-d834-4c0f-a2b2-89817b15f1e9" (UID: "98f1f260-d834-4c0f-a2b2-89817b15f1e9"). InnerVolumeSpecName "kube-api-access-sbmfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:20.905761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:20.905203    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbmfs\" (UniqueName: \"kubernetes.io/projected/98f1f260-d834-4c0f-a2b2-89817b15f1e9-kube-api-access-sbmfs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:20.905761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:20.905248    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98f1f260-d834-4c0f-a2b2-89817b15f1e9-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:21.175030 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:21.174947    5043 scope.go:122] "RemoveContainer" containerID="6ff7887b82f2093fa35a3a8e25dafdc854523d7a21407d568d9564348186ab13"
Oct 30 19:34:21.189057 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:21.188908    5043 scope.go:122] "RemoveContainer" containerID="2ba520ef9b55c000e0909de2296a3242fedc78015a21c5202485a2f4e35816f2"
Oct 30 19:34:21.203030 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:21.203003    5043 scope.go:122] "RemoveContainer" containerID="5073573f1a0f66acb54784b6f0c28f4568be3c3dfb324d957eb89bd0be74d590"
Oct 30 19:34:21.214547 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:21.214523    5043 scope.go:122] "RemoveContainer" containerID="576b8992df3a51a7e1e81961ed4457b7c19ce899d71e62254f117e2e667c6d61"
Oct 30 19:34:21.226161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:21.226140    5043 scope.go:122] "RemoveContainer" containerID="c6ed548dd294d66f989d76d4d629835e6f7829db8444c3d77b7c867b2f139eb9"
Oct 30 19:34:21.236989 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:21.236969    5043 scope.go:122] "RemoveContainer" containerID="e27482af84e53576b7cd04bbb54aa95f797bd6f6f9a98839b35bf514a6ad8b9f"
Oct 30 19:34:21.247471 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:21.247453    5043 scope.go:122] "RemoveContainer" containerID="0df51b9498fd8afead5fe3f409db6fe5a704ad1c23fd728f6b8a8743c7125ebd"
Oct 30 19:34:21.259076 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:21.259053    5043 scope.go:122] "RemoveContainer" containerID="d3d85cf9cf4c584e50884d055554c89d3952816e08824ef0599589970d8fca5d"
Oct 30 19:34:21.270680 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:21.270654    5043 scope.go:122] "RemoveContainer" containerID="0e3745aaa570430063acdc892881882fe243c5e1d01b20454c13635a3130e9b8"
Oct 30 19:34:21.283989 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:21.283964    5043 scope.go:122] "RemoveContainer" containerID="9d33234436200af4ace587731a37c1290714fbc4785d4fdf4c1dda07ed4906c3"
Oct 30 19:34:21.301222 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:21.301200    5043 scope.go:122] "RemoveContainer" containerID="180b850d30e20cf38e85da4c96a355d45fc4b5fa2ac0afef708bf85b376f7f2d"
Oct 30 19:34:21.312874 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:21.312851    5043 scope.go:122] "RemoveContainer" containerID="777069f4d4eb11eeacc5860aa9bc839226e15109dd398d9f7d758896289eb7ba"
Oct 30 19:34:21.324494 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:21.324468    5043 scope.go:122] "RemoveContainer" containerID="2839512c12bdadf5b9a12229d9517f4f156dfadd6e531053fe955db4025c46e8"
Oct 30 19:34:21.336945 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:21.335315    5043 scope.go:122] "RemoveContainer" containerID="901f369a951a610ce2016f8d213d2c63629cab4f9bcaf497ed99281c383f6611"
Oct 30 19:34:21.349941 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:21.349918    5043 scope.go:122] "RemoveContainer" containerID="3aa0d45a656082438385ec881f85cd46a257b5ed09e89938b4cde963ace10a45"
Oct 30 19:34:21.363087 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:21.363064    5043 scope.go:122] "RemoveContainer" containerID="77480000ae1c5856dae25601d7feb64238e6ffb3619ddd3ff28beb70941d7b77"
Oct 30 19:34:21.377546 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:21.377519    5043 scope.go:122] "RemoveContainer" containerID="eb49954fe25115b44fae9855f478d44a1839f3f46f0db4a440de9202b7315f3f"
Oct 30 19:34:21.391008 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:21.390988    5043 scope.go:122] "RemoveContainer" containerID="3b8f9dd58831d2329512bc76b88e9582cd4b0f9447d21da7a60354d543f1483f"
Oct 30 19:34:21.404399 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:21.404375    5043 scope.go:122] "RemoveContainer" containerID="f0ad75a101159fbb67097c30fb0703575d1c8342e584b058ac6486a0128811fa"
Oct 30 19:34:21.557335 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:21.557308    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c67e12-75v4f"
Oct 30 19:34:21.557432 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:21.557336    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8c67e12-75v4f" event={"ID":"98f1f260-d834-4c0f-a2b2-89817b15f1e9","Type":"ContainerDied","Data":"63484a74d484e288f35462adbfb560c0469f4d9f8e1380db6512625d03f8cc38"}
Oct 30 19:34:21.557496 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:21.557438    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63484a74d484e288f35462adbfb560c0469f4d9f8e1380db6512625d03f8cc38"
Oct 30 19:34:24.210998 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:24.210963    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fd9a937-j8w5c"]
Oct 30 19:34:24.211634 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:24.211614    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2fd9a937-j8w5c"]
Oct 30 19:34:24.665931 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:24.665841    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0" path="/var/lib/kubelet/pods/38eb9929-a6b3-45f6-8fa0-c8a26abf5ee0/volumes"
Oct 30 19:34:24.985921 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:34:24.985888    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:34:25.032724 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:25.032695    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66d1bdfd-n8fks"]
Oct 30 19:34:25.038895 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:25.038869    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66d1bdfd-n8fks"]
Oct 30 19:34:26.691638 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:26.684261    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01a3afe0-c101-49e1-ae81-66fff2d8bedf" path="/var/lib/kubelet/pods/01a3afe0-c101-49e1-ae81-66fff2d8bedf/volumes"
Oct 30 19:34:31.091759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:31.090797    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ee18dc2-cmntm"]
Oct 30 19:34:31.175761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:31.175648    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ee18dc2-cmntm"]
Oct 30 19:34:31.824764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:31.820846    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:34:31.893292 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:31.892714    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:34:31.910695 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:31.909756    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:34:31.983913 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:31.983866    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:34:32.767767 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:32.766860    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9c119ec-7866-483b-a775-e3438b62b386" path="/var/lib/kubelet/pods/b9c119ec-7866-483b-a775-e3438b62b386/volumes"
Oct 30 19:34:32.910509 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:32.910462    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:34:32.911293 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:32.911268    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:34:33.040889 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:33.036128    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:34:33.126774 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:33.125798    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:34:33.126982 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:33.086035    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:34:33.162486 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:33.162440    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:34:33.434253 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:33.434130    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:34:33.446883 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:33.446840    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:34:33.827796 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:33.810322    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded"
Oct 30 19:34:33.904879 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:33.884039    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": context deadline exceeded"
Oct 30 19:34:33.920665 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:33.918258    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:34:33.982456 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:33.981940    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:34:34.125427 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:34.122147    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:34:34.125427 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:34.125210    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:34:34.268776 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:34.227653    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:34:34.268776 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:34.247290    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:34:34.299139 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:34.299089    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:34:34.630799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:34.613424    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:34:34.797887 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:34.797842    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:34:34.823777 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:34.803460    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:34:34.823777 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:34.803560    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:34:34.998762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:34.983181    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:34:35.002241 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:35.002200    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:34:35.073347 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:35.073299    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:34:35.148881 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:35.148837    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:34:35.151183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:35.150388    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:34:35.445831 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:35.301670    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2d34376-c2wxr"]
Oct 30 19:34:35.523765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:35.462532    5043 cpu_manager.go:414] "containerMap: removing container" podUID="98f1f260-d834-4c0f-a2b2-89817b15f1e9" containerName="revoke-keys"
Oct 30 19:34:35.523765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:35.512236    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98f1f260-d834-4c0f-a2b2-89817b15f1e9" containerName="revoke-keys"
Oct 30 19:34:35.523765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:35.512644    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="98f1f260-d834-4c0f-a2b2-89817b15f1e9" containerName="revoke-keys"
Oct 30 19:34:35.540768 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:35.317374    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:34:35.600688 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:35.564453    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"
Oct 30 19:34:35.600688 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:35.564492    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m"]
Oct 30 19:34:35.600688 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:35.564551    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:34:35.600688 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:35.564588    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:34:35.600688 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:35.564625    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:34:35.600688 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:35.564658    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:34:35.600688 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:35.564687    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"
Oct 30 19:34:35.600688 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:35.594868    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m" podUID="a5a07d80-0678-4d80-bbc6-d17cb0ba35cf" containerName="istio-proxy" containerID="cri-o://fa6e01ab36eebcbfefe09cbbdf10ad3b6d4c66427d4b6abf0cbe86964415fd85" gracePeriod=30
Oct 30 19:34:35.614772 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:35.606810    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2d34376-c2wxr"
Oct 30 19:34:35.636759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:35.626376    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2d34376-c2wxr"]
Oct 30 19:34:35.636759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:35.631450    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:34:35.636759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:35.631488    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:34:35.742812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:35.711326    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:34:35.742812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:35.712996    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2db820e0-2c23-42aa-96f4-184da2fe7141-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2d34376-c2wxr\" (UID: \"2db820e0-2c23-42aa-96f4-184da2fe7141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2d34376-c2wxr"
Oct 30 19:34:35.742812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:35.713093    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwqp9\" (UniqueName: \"kubernetes.io/projected/2db820e0-2c23-42aa-96f4-184da2fe7141-kube-api-access-nwqp9\") pod \"maas-api-revoke-sub-d2d34376-c2wxr\" (UID: \"2db820e0-2c23-42aa-96f4-184da2fe7141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2d34376-c2wxr"
Oct 30 19:34:35.742812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:35.713443    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2db820e0-2c23-42aa-96f4-184da2fe7141-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2d34376-c2wxr\" (UID: \"2db820e0-2c23-42aa-96f4-184da2fe7141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2d34376-c2wxr"
Oct 30 19:34:35.742812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:35.733984    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:34:35.742812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:35.734073    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:34:35.840762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:35.839308    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwqp9\" (UniqueName: \"kubernetes.io/projected/2db820e0-2c23-42aa-96f4-184da2fe7141-kube-api-access-nwqp9\") pod \"maas-api-revoke-sub-d2d34376-c2wxr\" (UID: \"2db820e0-2c23-42aa-96f4-184da2fe7141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2d34376-c2wxr"
Oct 30 19:34:35.840762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:35.839597    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2db820e0-2c23-42aa-96f4-184da2fe7141-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2d34376-c2wxr\" (UID: \"2db820e0-2c23-42aa-96f4-184da2fe7141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2d34376-c2wxr"
Oct 30 19:34:35.840762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:35.839886    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2db820e0-2c23-42aa-96f4-184da2fe7141-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2d34376-c2wxr\" (UID: \"2db820e0-2c23-42aa-96f4-184da2fe7141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2d34376-c2wxr"
Oct 30 19:34:35.865763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:35.843259    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2db820e0-2c23-42aa-96f4-184da2fe7141-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2d34376-c2wxr\" (UID: \"2db820e0-2c23-42aa-96f4-184da2fe7141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2d34376-c2wxr"
Oct 30 19:34:36.114999 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:36.050940    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d51b507-prwfs"]
Oct 30 19:34:36.114999 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:36.050993    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d51b507-prwfs"]
Oct 30 19:34:36.160857 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:36.129954    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:34:36.222643 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:36.218279    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:34:36.326768 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:36.325643    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:34:36.347462 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:36.347427    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2db820e0-2c23-42aa-96f4-184da2fe7141-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2d34376-c2wxr\" (UID: \"2db820e0-2c23-42aa-96f4-184da2fe7141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2d34376-c2wxr"
Oct 30 19:34:36.610760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:36.607478    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwqp9\" (UniqueName: \"kubernetes.io/projected/2db820e0-2c23-42aa-96f4-184da2fe7141-kube-api-access-nwqp9\") pod \"maas-api-revoke-sub-d2d34376-c2wxr\" (UID: \"2db820e0-2c23-42aa-96f4-184da2fe7141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2d34376-c2wxr"
Oct 30 19:34:36.610760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:36.609703    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:34:36.637760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:36.631899    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:34:36.672293 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:36.672263    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2d34376-c2wxr"
Oct 30 19:34:37.133458 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:37.133417    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47cab750-d31e-4168-9817-024dc97aa9f0" path="/var/lib/kubelet/pods/47cab750-d31e-4168-9817-024dc97aa9f0/volumes"
Oct 30 19:34:37.134101 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:37.134047    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:34:37.225043 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:37.224982    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34ac9afd-6f5lh"]
Oct 30 19:34:37.234429 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:37.234401    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46b354b3-f2tds"]
Oct 30 19:34:37.242624 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:37.242563    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34ac9afd-6f5lh"]
Oct 30 19:34:37.249009 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:37.248708    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46b354b3-f2tds"]
Oct 30 19:34:37.405338 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:37.404171    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2d34376-c2wxr"]
Oct 30 19:34:37.408559 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:34:37.408529    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2db820e0_2c23_42aa_96f4_184da2fe7141.slice/crio-149a684be7615b96d13c57c6da1304255ad90c17b738e9df60ab8c7ed3d2eab7 WatchSource:0}: Error finding container 149a684be7615b96d13c57c6da1304255ad90c17b738e9df60ab8c7ed3d2eab7: Status 404 returned error can't find the container with id 149a684be7615b96d13c57c6da1304255ad90c17b738e9df60ab8c7ed3d2eab7
Oct 30 19:34:38.522781 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:38.520610    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2d34376-c2wxr" event={"ID":"2db820e0-2c23-42aa-96f4-184da2fe7141","Type":"ContainerStarted","Data":"2176ca0d283de4eb2ef7baaca8a15906367fa321a97f6798f79337bbb0ddf971"}
Oct 30 19:34:38.522781 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:38.520681    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2d34376-c2wxr" event={"ID":"2db820e0-2c23-42aa-96f4-184da2fe7141","Type":"ContainerStarted","Data":"149a684be7615b96d13c57c6da1304255ad90c17b738e9df60ab8c7ed3d2eab7"}
Oct 30 19:34:38.542544 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:38.542486    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2d34376-c2wxr" podStartSLOduration=5.542466535 podStartE2EDuration="5.542466535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:38.538643261 +0000 UTC m=+1098.049703226" watchObservedRunningTime="2026-10-30 19:34:38.542466535 +0000 UTC m=+1098.053526503"
Oct 30 19:34:38.649371 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:38.649335    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db1bb288-ae84-4f59-bf73-2152e976377b" path="/var/lib/kubelet/pods/db1bb288-ae84-4f59-bf73-2152e976377b/volumes"
Oct 30 19:34:38.650162 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:38.650137    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fda28b52-1ec6-4771-b1ae-d02a0598af42" path="/var/lib/kubelet/pods/fda28b52-1ec6-4771-b1ae-d02a0598af42/volumes"
Oct 30 19:34:39.030867 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:39.030830    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00c784c1-7rgrb"]
Oct 30 19:34:39.037780 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:39.037726    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00c784c1-7rgrb"]
Oct 30 19:34:39.537187 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:39.537156    5043 generic.go:356] "Generic (PLEG): container finished" podID="2db820e0-2c23-42aa-96f4-184da2fe7141" containerID="2176ca0d283de4eb2ef7baaca8a15906367fa321a97f6798f79337bbb0ddf971" exitCode=0
Oct 30 19:34:39.537612 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:39.537197    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2d34376-c2wxr" event={"ID":"2db820e0-2c23-42aa-96f4-184da2fe7141","Type":"ContainerDied","Data":"2176ca0d283de4eb2ef7baaca8a15906367fa321a97f6798f79337bbb0ddf971"}
Oct 30 19:34:40.675972 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:40.675934    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34ada6b0-b808-4f1d-a770-4ce3219722ca" path="/var/lib/kubelet/pods/34ada6b0-b808-4f1d-a770-4ce3219722ca/volumes"
Oct 30 19:34:40.875845 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:40.875679    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2d34376-c2wxr"
Oct 30 19:34:40.975976 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:40.975879    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2db820e0-2c23-42aa-96f4-184da2fe7141-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2db820e0-2c23-42aa-96f4-184da2fe7141-maas-api-openshift-ca\") pod \"2db820e0-2c23-42aa-96f4-184da2fe7141\" (UID: \"2db820e0-2c23-42aa-96f4-184da2fe7141\") "
Oct 30 19:34:40.976374 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:40.976335    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2db820e0-2c23-42aa-96f4-184da2fe7141-maas-api-openshift-ca" pod "2db820e0-2c23-42aa-96f4-184da2fe7141" (UID: "2db820e0-2c23-42aa-96f4-184da2fe7141"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:40.976657 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:40.976629    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2db820e0-2c23-42aa-96f4-184da2fe7141-kube-api-access-nwqp9\" (UniqueName: \"kubernetes.io/projected/2db820e0-2c23-42aa-96f4-184da2fe7141-kube-api-access-nwqp9\") pod \"2db820e0-2c23-42aa-96f4-184da2fe7141\" (UID: \"2db820e0-2c23-42aa-96f4-184da2fe7141\") "
Oct 30 19:34:40.976905 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:40.976687    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2db820e0-2c23-42aa-96f4-184da2fe7141-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2db820e0-2c23-42aa-96f4-184da2fe7141-maas-api-xks-ca\") pod \"2db820e0-2c23-42aa-96f4-184da2fe7141\" (UID: \"2db820e0-2c23-42aa-96f4-184da2fe7141\") "
Oct 30 19:34:40.978644 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:40.978616    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2db820e0-2c23-42aa-96f4-184da2fe7141-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:40.982206 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:40.982165    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2db820e0-2c23-42aa-96f4-184da2fe7141-maas-api-xks-ca" pod "2db820e0-2c23-42aa-96f4-184da2fe7141" (UID: "2db820e0-2c23-42aa-96f4-184da2fe7141"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:34:40.984471 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:40.984431    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2db820e0-2c23-42aa-96f4-184da2fe7141-kube-api-access-nwqp9" pod "2db820e0-2c23-42aa-96f4-184da2fe7141" (UID: "2db820e0-2c23-42aa-96f4-184da2fe7141"). InnerVolumeSpecName "kube-api-access-nwqp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:41.060266 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.060219    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m"
Oct 30 19:34:41.081868 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.081424    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2db820e0-2c23-42aa-96f4-184da2fe7141-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:41.081868 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.081465    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwqp9\" (UniqueName: \"kubernetes.io/projected/2db820e0-2c23-42aa-96f4-184da2fe7141-kube-api-access-nwqp9\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:41.182614 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.182515    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-istiod-ca-cert\") pod \"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf\" (UID: \"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf\") "
Oct 30 19:34:41.182802 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.182642    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-workload-certs\") pod \"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf\" (UID: \"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf\") "
Oct 30 19:34:41.182802 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.182683    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-credential-socket\") pod \"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf\" (UID: \"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf\") "
Oct 30 19:34:41.182802 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.182771    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-istio-envoy\") pod \"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf\" (UID: \"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf\") "
Oct 30 19:34:41.182969 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.182827    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-kube-api-access-5dzf5\" (UniqueName: \"kubernetes.io/projected/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-kube-api-access-5dzf5\") pod \"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf\" (UID: \"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf\") "
Oct 30 19:34:41.183045 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.183023    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-istio-token\" (UniqueName: \"kubernetes.io/projected/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-istio-token\") pod \"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf\" (UID: \"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf\") "
Oct 30 19:34:41.183284 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.183252    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-workload-certs" pod "a5a07d80-0678-4d80-bbc6-d17cb0ba35cf" (UID: "a5a07d80-0678-4d80-bbc6-d17cb0ba35cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:41.183382 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.183332    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-credential-socket" pod "a5a07d80-0678-4d80-bbc6-d17cb0ba35cf" (UID: "a5a07d80-0678-4d80-bbc6-d17cb0ba35cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:41.183382 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.183349    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-istiod-ca-cert" pod "a5a07d80-0678-4d80-bbc6-d17cb0ba35cf" (UID: "a5a07d80-0678-4d80-bbc6-d17cb0ba35cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:41.183498 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.183458    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-istio-podinfo\") pod \"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf\" (UID: \"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf\") "
Oct 30 19:34:41.183554 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.183538    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-workload-socket\") pod \"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf\" (UID: \"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf\") "
Oct 30 19:34:41.183688 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.183671    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-istio-data\") pod \"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf\" (UID: \"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf\") "
Oct 30 19:34:41.184473 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.184446    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-workload-socket" pod "a5a07d80-0678-4d80-bbc6-d17cb0ba35cf" (UID: "a5a07d80-0678-4d80-bbc6-d17cb0ba35cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:41.185093 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.185068    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:41.185209 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.185097    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:41.185209 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.185114    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:41.185209 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.185127    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:41.185395 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.185331    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-istio-data" pod "a5a07d80-0678-4d80-bbc6-d17cb0ba35cf" (UID: "a5a07d80-0678-4d80-bbc6-d17cb0ba35cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:41.187053 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.187003    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-istio-envoy" pod "a5a07d80-0678-4d80-bbc6-d17cb0ba35cf" (UID: "a5a07d80-0678-4d80-bbc6-d17cb0ba35cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:41.187176 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.187150    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-kube-api-access-5dzf5" pod "a5a07d80-0678-4d80-bbc6-d17cb0ba35cf" (UID: "a5a07d80-0678-4d80-bbc6-d17cb0ba35cf"). InnerVolumeSpecName "kube-api-access-5dzf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:41.189981 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.189938    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-istio-token" pod "a5a07d80-0678-4d80-bbc6-d17cb0ba35cf" (UID: "a5a07d80-0678-4d80-bbc6-d17cb0ba35cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:41.190478 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.190441    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-istio-podinfo" pod "a5a07d80-0678-4d80-bbc6-d17cb0ba35cf" (UID: "a5a07d80-0678-4d80-bbc6-d17cb0ba35cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:34:41.210174 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.210135    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c7880b-c46dcb8b7-dkz2k"]
Oct 30 19:34:41.211746 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.211710    5043 cpu_manager.go:414] "containerMap: removing container" podUID="a5a07d80-0678-4d80-bbc6-d17cb0ba35cf" containerName="istio-proxy"
Oct 30 19:34:41.211862 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.211757    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5a07d80-0678-4d80-bbc6-d17cb0ba35cf" containerName="istio-proxy"
Oct 30 19:34:41.211862 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.211813    5043 cpu_manager.go:414] "containerMap: removing container" podUID="a5a07d80-0678-4d80-bbc6-d17cb0ba35cf" containerName="istio-proxy"
Oct 30 19:34:41.211862 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.211821    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5a07d80-0678-4d80-bbc6-d17cb0ba35cf" containerName="istio-proxy"
Oct 30 19:34:41.211862 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.211852    5043 cpu_manager.go:414] "containerMap: removing container" podUID="2db820e0-2c23-42aa-96f4-184da2fe7141" containerName="revoke-keys"
Oct 30 19:34:41.211862 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.211860    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2db820e0-2c23-42aa-96f4-184da2fe7141" containerName="revoke-keys"
Oct 30 19:34:41.212067 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.212052    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5a07d80-0678-4d80-bbc6-d17cb0ba35cf" containerName="istio-proxy"
Oct 30 19:34:41.212104 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.212092    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="2db820e0-2c23-42aa-96f4-184da2fe7141" containerName="revoke-keys"
Oct 30 19:34:41.221954 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.221156    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c7880b-c46dcb8b7-dkz2k"
Oct 30 19:34:41.223868 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.223834    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c7880b-c46dcb8b7-dkz2k"]
Oct 30 19:34:41.230576 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.230540    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-c7880b-dockercfg-l2v2n\""
Oct 30 19:34:41.232602 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.232569    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c7880b-d5b8b5f94-gwr52"]
Oct 30 19:34:41.233791 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.233761    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5a07d80-0678-4d80-bbc6-d17cb0ba35cf" containerName="istio-proxy"
Oct 30 19:34:41.238526 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.238501    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c7880b-d5b8b5f94-gwr52"
Oct 30 19:34:41.262635 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.262551    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c7880b-d5b8b5f94-gwr52"]
Oct 30 19:34:41.293190 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.284926    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-c7880b\""
Oct 30 19:34:41.293190 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.288624    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:41.293190 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.288660    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dzf5\" (UniqueName: \"kubernetes.io/projected/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-kube-api-access-5dzf5\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:41.293190 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.288681    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:41.293190 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.288695    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:41.293190 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.288709    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:41.396486 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.392315    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p65vz\" (UniqueName: \"kubernetes.io/projected/b7ee29a5-f392-4c4f-907b-dac7c0140f6c-kube-api-access-p65vz\") pod \"payload-pre-processing-e2e-api-a-c7880b-c46dcb8b7-dkz2k\" (UID: \"b7ee29a5-f392-4c4f-907b-dac7c0140f6c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c7880b-c46dcb8b7-dkz2k"
Oct 30 19:34:41.396930 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.396874    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r74lq\" (UniqueName: \"kubernetes.io/projected/9d198eae-9067-4789-b6b1-b8827ca742ab-kube-api-access-r74lq\") pod \"payload-processing-e2e-api-a-c7880b-d5b8b5f94-gwr52\" (UID: \"9d198eae-9067-4789-b6b1-b8827ca742ab\") " pod="openshift-ingress/payload-processing-e2e-api-a-c7880b-d5b8b5f94-gwr52"
Oct 30 19:34:41.397360 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.397331    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d198eae-9067-4789-b6b1-b8827ca742ab-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c7880b-d5b8b5f94-gwr52\" (UID: \"9d198eae-9067-4789-b6b1-b8827ca742ab\") " pod="openshift-ingress/payload-processing-e2e-api-a-c7880b-d5b8b5f94-gwr52"
Oct 30 19:34:41.397912 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.397886    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7ee29a5-f392-4c4f-907b-dac7c0140f6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c7880b-c46dcb8b7-dkz2k\" (UID: \"b7ee29a5-f392-4c4f-907b-dac7c0140f6c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c7880b-c46dcb8b7-dkz2k"
Oct 30 19:34:41.501181 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.500133    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7ee29a5-f392-4c4f-907b-dac7c0140f6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c7880b-c46dcb8b7-dkz2k\" (UID: \"b7ee29a5-f392-4c4f-907b-dac7c0140f6c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c7880b-c46dcb8b7-dkz2k"
Oct 30 19:34:41.501181 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.500285    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p65vz\" (UniqueName: \"kubernetes.io/projected/b7ee29a5-f392-4c4f-907b-dac7c0140f6c-kube-api-access-p65vz\") pod \"payload-pre-processing-e2e-api-a-c7880b-c46dcb8b7-dkz2k\" (UID: \"b7ee29a5-f392-4c4f-907b-dac7c0140f6c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c7880b-c46dcb8b7-dkz2k"
Oct 30 19:34:41.501181 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.500325    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r74lq\" (UniqueName: \"kubernetes.io/projected/9d198eae-9067-4789-b6b1-b8827ca742ab-kube-api-access-r74lq\") pod \"payload-processing-e2e-api-a-c7880b-d5b8b5f94-gwr52\" (UID: \"9d198eae-9067-4789-b6b1-b8827ca742ab\") " pod="openshift-ingress/payload-processing-e2e-api-a-c7880b-d5b8b5f94-gwr52"
Oct 30 19:34:41.501181 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.500451    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d198eae-9067-4789-b6b1-b8827ca742ab-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c7880b-d5b8b5f94-gwr52\" (UID: \"9d198eae-9067-4789-b6b1-b8827ca742ab\") " pod="openshift-ingress/payload-processing-e2e-api-a-c7880b-d5b8b5f94-gwr52"
Oct 30 19:34:41.506857 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.505902    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d198eae-9067-4789-b6b1-b8827ca742ab-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c7880b-d5b8b5f94-gwr52\" (UID: \"9d198eae-9067-4789-b6b1-b8827ca742ab\") " pod="openshift-ingress/payload-processing-e2e-api-a-c7880b-d5b8b5f94-gwr52"
Oct 30 19:34:41.516768 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.514357    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7ee29a5-f392-4c4f-907b-dac7c0140f6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c7880b-c46dcb8b7-dkz2k\" (UID: \"b7ee29a5-f392-4c4f-907b-dac7c0140f6c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c7880b-c46dcb8b7-dkz2k"
Oct 30 19:34:41.557600 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.556319    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p65vz\" (UniqueName: \"kubernetes.io/projected/b7ee29a5-f392-4c4f-907b-dac7c0140f6c-kube-api-access-p65vz\") pod \"payload-pre-processing-e2e-api-a-c7880b-c46dcb8b7-dkz2k\" (UID: \"b7ee29a5-f392-4c4f-907b-dac7c0140f6c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c7880b-c46dcb8b7-dkz2k"
Oct 30 19:34:41.566073 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.566035    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r74lq\" (UniqueName: \"kubernetes.io/projected/9d198eae-9067-4789-b6b1-b8827ca742ab-kube-api-access-r74lq\") pod \"payload-processing-e2e-api-a-c7880b-d5b8b5f94-gwr52\" (UID: \"9d198eae-9067-4789-b6b1-b8827ca742ab\") " pod="openshift-ingress/payload-processing-e2e-api-a-c7880b-d5b8b5f94-gwr52"
Oct 30 19:34:41.608756 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.608692    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c7880b-d5b8b5f94-gwr52"
Oct 30 19:34:41.620123 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.610349    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c7880b-c46dcb8b7-dkz2k"
Oct 30 19:34:41.701621 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.685662    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2d34376-c2wxr" event={"ID":"2db820e0-2c23-42aa-96f4-184da2fe7141","Type":"ContainerDied","Data":"149a684be7615b96d13c57c6da1304255ad90c17b738e9df60ab8c7ed3d2eab7"}
Oct 30 19:34:41.701621 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.685729    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="149a684be7615b96d13c57c6da1304255ad90c17b738e9df60ab8c7ed3d2eab7"
Oct 30 19:34:41.701621 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.685835    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2d34376-c2wxr"
Oct 30 19:34:41.760962 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.760782    5043 generic.go:356] "Generic (PLEG): container finished" podID="a5a07d80-0678-4d80-bbc6-d17cb0ba35cf" containerID="fa6e01ab36eebcbfefe09cbbdf10ad3b6d4c66427d4b6abf0cbe86964415fd85" exitCode=0
Oct 30 19:34:41.761896 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.761128    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m" event={"ID":"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf","Type":"ContainerDied","Data":"fa6e01ab36eebcbfefe09cbbdf10ad3b6d4c66427d4b6abf0cbe86964415fd85"}
Oct 30 19:34:41.761896 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.761186    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m" event={"ID":"a5a07d80-0678-4d80-bbc6-d17cb0ba35cf","Type":"ContainerDied","Data":"63e499e88f780de67383fcb53ccc9f1f60f2b16df68fca5bcd307d3d5333a94f"}
Oct 30 19:34:41.761896 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.761220    5043 scope.go:122] "RemoveContainer" containerID="fa6e01ab36eebcbfefe09cbbdf10ad3b6d4c66427d4b6abf0cbe86964415fd85"
Oct 30 19:34:41.761896 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.761397    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m"
Oct 30 19:34:41.825845 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.825566    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m"]
Oct 30 19:34:41.835398 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.835363    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2fd9a937-openshift-default-76594d45b5-55g7m"]
Oct 30 19:34:41.874685 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.874656    5043 scope.go:122] "RemoveContainer" containerID="bc800dccc98b72bcbb1f47fb11672198a09ae67496bf6bf35a8766e88235a0f1"
Oct 30 19:34:41.908060 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.897662    5043 scope.go:122] "RemoveContainer" containerID="fa6e01ab36eebcbfefe09cbbdf10ad3b6d4c66427d4b6abf0cbe86964415fd85"
Oct 30 19:34:41.908060 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:34:41.898132    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa6e01ab36eebcbfefe09cbbdf10ad3b6d4c66427d4b6abf0cbe86964415fd85\": container with ID starting with fa6e01ab36eebcbfefe09cbbdf10ad3b6d4c66427d4b6abf0cbe86964415fd85 not found: ID does not exist" containerID="fa6e01ab36eebcbfefe09cbbdf10ad3b6d4c66427d4b6abf0cbe86964415fd85"
Oct 30 19:34:41.908060 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.898176    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa6e01ab36eebcbfefe09cbbdf10ad3b6d4c66427d4b6abf0cbe86964415fd85"} err="failed to get container status \"fa6e01ab36eebcbfefe09cbbdf10ad3b6d4c66427d4b6abf0cbe86964415fd85\": rpc error: code = NotFound desc = could not find container \"fa6e01ab36eebcbfefe09cbbdf10ad3b6d4c66427d4b6abf0cbe86964415fd85\": container with ID starting with fa6e01ab36eebcbfefe09cbbdf10ad3b6d4c66427d4b6abf0cbe86964415fd85 not found: ID does not exist"
Oct 30 19:34:41.908060 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.898196    5043 scope.go:122] "RemoveContainer" containerID="bc800dccc98b72bcbb1f47fb11672198a09ae67496bf6bf35a8766e88235a0f1"
Oct 30 19:34:41.908060 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:34:41.898478    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc800dccc98b72bcbb1f47fb11672198a09ae67496bf6bf35a8766e88235a0f1\": container with ID starting with bc800dccc98b72bcbb1f47fb11672198a09ae67496bf6bf35a8766e88235a0f1 not found: ID does not exist" containerID="bc800dccc98b72bcbb1f47fb11672198a09ae67496bf6bf35a8766e88235a0f1"
Oct 30 19:34:41.908060 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:41.898506    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc800dccc98b72bcbb1f47fb11672198a09ae67496bf6bf35a8766e88235a0f1"} err="failed to get container status \"bc800dccc98b72bcbb1f47fb11672198a09ae67496bf6bf35a8766e88235a0f1\": rpc error: code = NotFound desc = could not find container \"bc800dccc98b72bcbb1f47fb11672198a09ae67496bf6bf35a8766e88235a0f1\": container with ID starting with bc800dccc98b72bcbb1f47fb11672198a09ae67496bf6bf35a8766e88235a0f1 not found: ID does not exist"
Oct 30 19:34:42.100590 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:42.090930    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-799546ec-lnflr"]
Oct 30 19:34:42.107091 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:42.107064    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-799546ec-lnflr"]
Oct 30 19:34:42.238949 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:34:42.238903    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d198eae_9067_4789_b6b1_b8827ca742ab.slice/crio-340842a1759bae884399dd726e072fe068b0d0a336dfbc54dab9bed50fc0fb87 WatchSource:0}: Error finding container 340842a1759bae884399dd726e072fe068b0d0a336dfbc54dab9bed50fc0fb87: Status 404 returned error can't find the container with id 340842a1759bae884399dd726e072fe068b0d0a336dfbc54dab9bed50fc0fb87
Oct 30 19:34:42.241568 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:42.241538    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c7880b-d5b8b5f94-gwr52"]
Oct 30 19:34:42.319345 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:34:42.319316    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7ee29a5_f392_4c4f_907b_dac7c0140f6c.slice/crio-02587fe980b65c70fba0bbb589ecbb5d342edda7e0800bf69da9daaa47180b41 WatchSource:0}: Error finding container 02587fe980b65c70fba0bbb589ecbb5d342edda7e0800bf69da9daaa47180b41: Status 404 returned error can't find the container with id 02587fe980b65c70fba0bbb589ecbb5d342edda7e0800bf69da9daaa47180b41
Oct 30 19:34:42.319791 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:42.319762    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c7880b-c46dcb8b7-dkz2k"]
Oct 30 19:34:42.650514 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:42.650432    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51267af2-5e1b-438e-b1be-25f8f5ee2d9f" path="/var/lib/kubelet/pods/51267af2-5e1b-438e-b1be-25f8f5ee2d9f/volumes"
Oct 30 19:34:42.651091 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:42.651067    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5a07d80-0678-4d80-bbc6-d17cb0ba35cf" path="/var/lib/kubelet/pods/a5a07d80-0678-4d80-bbc6-d17cb0ba35cf/volumes"
Oct 30 19:34:42.822806 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:42.822715    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c7880b-c46dcb8b7-dkz2k" event={"ID":"b7ee29a5-f392-4c4f-907b-dac7c0140f6c","Type":"ContainerStarted","Data":"1a739e75c29e65878454639cd97bc127d806eb8ccc45087f347ce0f17a32dd95"}
Oct 30 19:34:42.823401 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:42.823374    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c7880b-c46dcb8b7-dkz2k" event={"ID":"b7ee29a5-f392-4c4f-907b-dac7c0140f6c","Type":"ContainerStarted","Data":"02587fe980b65c70fba0bbb589ecbb5d342edda7e0800bf69da9daaa47180b41"}
Oct 30 19:34:42.828191 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:42.828163    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c7880b-c46dcb8b7-dkz2k"
Oct 30 19:34:42.830268 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:42.830240    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c7880b-d5b8b5f94-gwr52" event={"ID":"9d198eae-9067-4789-b6b1-b8827ca742ab","Type":"ContainerStarted","Data":"0ca3205e20388759dd42f2e1f7f6acc2fb9f01853c0924c43ed5f69560fd6937"}
Oct 30 19:34:42.830464 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:42.830449    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-c7880b-d5b8b5f94-gwr52"
Oct 30 19:34:42.830582 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:42.830567    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c7880b-d5b8b5f94-gwr52" event={"ID":"9d198eae-9067-4789-b6b1-b8827ca742ab","Type":"ContainerStarted","Data":"340842a1759bae884399dd726e072fe068b0d0a336dfbc54dab9bed50fc0fb87"}
Oct 30 19:34:42.887899 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:42.887824    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c7880b-c46dcb8b7-dkz2k" podStartSLOduration=1.8878023640000001 podStartE2EDuration="1.887802364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:42.857523678 +0000 UTC m=+1102.368583644" watchObservedRunningTime="2026-10-30 19:34:42.887802364 +0000 UTC m=+1102.398862330"
Oct 30 19:34:42.889072 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:42.889012    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-c7880b-d5b8b5f94-gwr52" podStartSLOduration=1.88900171 podStartE2EDuration="1.88900171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:42.875622456 +0000 UTC m=+1102.386682422" watchObservedRunningTime="2026-10-30 19:34:42.88900171 +0000 UTC m=+1102.400061677"
Oct 30 19:34:43.318945 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:43.318898    5043 patch_prober.go:29] interesting pod/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:34:43.319189 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:43.318969    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc" podUID="b34f07ce-6edf-4493-b349-ad7916bb93aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:34:44.863265 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:44.863232    5043 generic.go:356] "Generic (PLEG): container finished" podID="b34f07ce-6edf-4493-b349-ad7916bb93aa" containerID="35d3bb6a619b8fd1b7a20cd5648be3a3d97cbbd1ee8170fd1ebc766131c7f752" exitCode=0
Oct 30 19:34:44.863701 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:44.863315    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc" event={"ID":"b34f07ce-6edf-4493-b349-ad7916bb93aa","Type":"ContainerDied","Data":"35d3bb6a619b8fd1b7a20cd5648be3a3d97cbbd1ee8170fd1ebc766131c7f752"}
Oct 30 19:34:44.864075 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:44.864056    5043 scope.go:122] "RemoveContainer" containerID="35d3bb6a619b8fd1b7a20cd5648be3a3d97cbbd1ee8170fd1ebc766131c7f752"
Oct 30 19:34:44.866430 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:44.866401    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:34:44.866550 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:44.866485    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:34:44.866550 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:44.866523    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:34:45.877350 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:45.877309    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc" event={"ID":"b34f07ce-6edf-4493-b349-ad7916bb93aa","Type":"ContainerStarted","Data":"d1b0d067fd0d220353deb56c12459de13b426a871771643c434535c21c1ae5ff"}
Oct 30 19:34:46.138815 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:46.135898    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc"
Oct 30 19:34:46.138815 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:46.136024    5043 patch_prober.go:29] interesting pod/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 30 19:34:46.138815 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:46.136080    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc" podUID="b34f07ce-6edf-4493-b349-ad7916bb93aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 30 19:34:47.055498 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:47.055463    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2b9fef5-vfw6s"]
Oct 30 19:34:47.127618 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:47.127268    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2b9fef5-vfw6s"]
Oct 30 19:34:47.152759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:47.145164    5043 patch_prober.go:29] interesting pod/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 30 19:34:47.152759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:47.145229    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc" podUID="b34f07ce-6edf-4493-b349-ad7916bb93aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 30 19:34:48.135371 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:48.135340    5043 patch_prober.go:29] interesting pod/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 30 19:34:48.135800 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:48.135398    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc" podUID="b34f07ce-6edf-4493-b349-ad7916bb93aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 30 19:34:48.648946 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:48.648912    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad59b8d8-2911-4fe8-b7cd-9090ff58590c" path="/var/lib/kubelet/pods/ad59b8d8-2911-4fe8-b7cd-9090ff58590c/volumes"
Oct 30 19:34:49.135577 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:49.135540    5043 patch_prober.go:29] interesting pod/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 30 19:34:49.136049 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:49.135603    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc" podUID="b34f07ce-6edf-4493-b349-ad7916bb93aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 30 19:34:50.136798 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:50.135522    5043 patch_prober.go:29] interesting pod/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 30 19:34:50.136798 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:50.135612    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc" podUID="b34f07ce-6edf-4493-b349-ad7916bb93aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 30 19:34:51.135861 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:51.135824    5043 patch_prober.go:29] interesting pod/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 30 19:34:51.136068 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:51.135894    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc" podUID="b34f07ce-6edf-4493-b349-ad7916bb93aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 30 19:34:52.325763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:52.323379    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc"
Oct 30 19:34:52.325763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:52.325137    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc"
Oct 30 19:34:52.334765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:52.331303    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc"
Oct 30 19:34:53.855682 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:53.855657    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-c7880b-d5b8b5f94-gwr52"
Oct 30 19:34:54.867754 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:54.867719    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c7880b-c46dcb8b7-dkz2k"
Oct 30 19:34:54.991377 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:34:54.991325    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:34:56.069619 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:56.069571    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a8a9a63-wh84h"]
Oct 30 19:34:56.078366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:56.078331    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8966f23c-llfvq"]
Oct 30 19:34:56.088226 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:56.088188    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a8a9a63-wh84h"]
Oct 30 19:34:56.096812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:56.096778    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8966f23c-llfvq"]
Oct 30 19:34:56.648051 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:56.648008    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52b13089-acb8-4eee-a734-04113162fca4" path="/var/lib/kubelet/pods/52b13089-acb8-4eee-a734-04113162fca4/volumes"
Oct 30 19:34:56.648710 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:56.648684    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb02228c-861e-41bb-b16e-31eb93072532" path="/var/lib/kubelet/pods/fb02228c-861e-41bb-b16e-31eb93072532/volumes"
Oct 30 19:34:58.031016 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:58.030983    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d797f569-bfs6w"]
Oct 30 19:34:58.037034 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:58.037001    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d797f569-bfs6w"]
Oct 30 19:34:58.695612 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:34:58.693031    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f911451a-2556-49ed-9778-7faa6549d341" path="/var/lib/kubelet/pods/f911451a-2556-49ed-9778-7faa6549d341/volumes"
Oct 30 19:35:10.056135 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:10.056042    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d9567a8-229rl"]
Oct 30 19:35:10.073728 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:10.073690    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d9567a8-229rl"]
Oct 30 19:35:10.648777 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:10.648730    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d89aa71b-2ecb-475d-999f-7ed12d08098c" path="/var/lib/kubelet/pods/d89aa71b-2ecb-475d-999f-7ed12d08098c/volumes"
Oct 30 19:35:11.262692 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.261195    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk"]
Oct 30 19:35:11.269999 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.269972    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk"
Oct 30 19:35:11.283166 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.283135    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk"]
Oct 30 19:35:11.285452 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.285426    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-95d2ac-openshift-default-dockercfg-p7jqq\""
Oct 30 19:35:11.424712 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.424668    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08cde99-b619-487f-b34f-3180ba5019be-credential-socket\") pod \"e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk\" (UID: \"c08cde99-b619-487f-b34f-3180ba5019be\") " pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk"
Oct 30 19:35:11.425041 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.425017    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c08cde99-b619-487f-b34f-3180ba5019be-istio-podinfo\") pod \"e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk\" (UID: \"c08cde99-b619-487f-b34f-3180ba5019be\") " pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk"
Oct 30 19:35:11.425284 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.425266    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7swqx\" (UniqueName: \"kubernetes.io/projected/c08cde99-b619-487f-b34f-3180ba5019be-kube-api-access-7swqx\") pod \"e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk\" (UID: \"c08cde99-b619-487f-b34f-3180ba5019be\") " pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk"
Oct 30 19:35:11.425415 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.425398    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c08cde99-b619-487f-b34f-3180ba5019be-istio-envoy\") pod \"e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk\" (UID: \"c08cde99-b619-487f-b34f-3180ba5019be\") " pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk"
Oct 30 19:35:11.431850 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.431815    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c08cde99-b619-487f-b34f-3180ba5019be-workload-certs\") pod \"e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk\" (UID: \"c08cde99-b619-487f-b34f-3180ba5019be\") " pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk"
Oct 30 19:35:11.432071 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.432054    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c08cde99-b619-487f-b34f-3180ba5019be-istio-token\") pod \"e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk\" (UID: \"c08cde99-b619-487f-b34f-3180ba5019be\") " pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk"
Oct 30 19:35:11.432263 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.432248    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08cde99-b619-487f-b34f-3180ba5019be-workload-socket\") pod \"e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk\" (UID: \"c08cde99-b619-487f-b34f-3180ba5019be\") " pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk"
Oct 30 19:35:11.432646 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.432627    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c08cde99-b619-487f-b34f-3180ba5019be-istio-data\") pod \"e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk\" (UID: \"c08cde99-b619-487f-b34f-3180ba5019be\") " pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk"
Oct 30 19:35:11.432829 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.432812    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c08cde99-b619-487f-b34f-3180ba5019be-istiod-ca-cert\") pod \"e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk\" (UID: \"c08cde99-b619-487f-b34f-3180ba5019be\") " pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk"
Oct 30 19:35:11.538096 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.538002    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7swqx\" (UniqueName: \"kubernetes.io/projected/c08cde99-b619-487f-b34f-3180ba5019be-kube-api-access-7swqx\") pod \"e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk\" (UID: \"c08cde99-b619-487f-b34f-3180ba5019be\") " pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk"
Oct 30 19:35:11.538350 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.538326    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c08cde99-b619-487f-b34f-3180ba5019be-istio-envoy\") pod \"e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk\" (UID: \"c08cde99-b619-487f-b34f-3180ba5019be\") " pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk"
Oct 30 19:35:11.538471 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.538455    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c08cde99-b619-487f-b34f-3180ba5019be-workload-certs\") pod \"e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk\" (UID: \"c08cde99-b619-487f-b34f-3180ba5019be\") " pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk"
Oct 30 19:35:11.538589 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.538574    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c08cde99-b619-487f-b34f-3180ba5019be-istio-token\") pod \"e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk\" (UID: \"c08cde99-b619-487f-b34f-3180ba5019be\") " pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk"
Oct 30 19:35:11.539521 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.538724    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08cde99-b619-487f-b34f-3180ba5019be-workload-socket\") pod \"e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk\" (UID: \"c08cde99-b619-487f-b34f-3180ba5019be\") " pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk"
Oct 30 19:35:11.539890 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.539869    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c08cde99-b619-487f-b34f-3180ba5019be-istio-data\") pod \"e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk\" (UID: \"c08cde99-b619-487f-b34f-3180ba5019be\") " pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk"
Oct 30 19:35:11.551887 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.551852    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c08cde99-b619-487f-b34f-3180ba5019be-istiod-ca-cert\") pod \"e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk\" (UID: \"c08cde99-b619-487f-b34f-3180ba5019be\") " pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk"
Oct 30 19:35:11.552170 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.552153    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08cde99-b619-487f-b34f-3180ba5019be-credential-socket\") pod \"e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk\" (UID: \"c08cde99-b619-487f-b34f-3180ba5019be\") " pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk"
Oct 30 19:35:11.552311 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.552298    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c08cde99-b619-487f-b34f-3180ba5019be-istio-podinfo\") pod \"e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk\" (UID: \"c08cde99-b619-487f-b34f-3180ba5019be\") " pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk"
Oct 30 19:35:11.586760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.573150    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c08cde99-b619-487f-b34f-3180ba5019be-istio-data\") pod \"e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk\" (UID: \"c08cde99-b619-487f-b34f-3180ba5019be\") " pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk"
Oct 30 19:35:11.586760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.573445    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08cde99-b619-487f-b34f-3180ba5019be-workload-socket\") pod \"e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk\" (UID: \"c08cde99-b619-487f-b34f-3180ba5019be\") " pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk"
Oct 30 19:35:11.586760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.574079    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c08cde99-b619-487f-b34f-3180ba5019be-istiod-ca-cert\") pod \"e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk\" (UID: \"c08cde99-b619-487f-b34f-3180ba5019be\") " pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk"
Oct 30 19:35:11.586760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.574324    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08cde99-b619-487f-b34f-3180ba5019be-credential-socket\") pod \"e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk\" (UID: \"c08cde99-b619-487f-b34f-3180ba5019be\") " pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk"
Oct 30 19:35:11.587618 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.587588    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c08cde99-b619-487f-b34f-3180ba5019be-workload-certs\") pod \"e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk\" (UID: \"c08cde99-b619-487f-b34f-3180ba5019be\") " pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk"
Oct 30 19:35:11.614044 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.614003    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c08cde99-b619-487f-b34f-3180ba5019be-istio-envoy\") pod \"e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk\" (UID: \"c08cde99-b619-487f-b34f-3180ba5019be\") " pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk"
Oct 30 19:35:11.619308 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.619241    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c08cde99-b619-487f-b34f-3180ba5019be-istio-token\") pod \"e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk\" (UID: \"c08cde99-b619-487f-b34f-3180ba5019be\") " pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk"
Oct 30 19:35:11.619687 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.619665    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c08cde99-b619-487f-b34f-3180ba5019be-istio-podinfo\") pod \"e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk\" (UID: \"c08cde99-b619-487f-b34f-3180ba5019be\") " pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk"
Oct 30 19:35:11.647799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.641467    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7swqx\" (UniqueName: \"kubernetes.io/projected/c08cde99-b619-487f-b34f-3180ba5019be-kube-api-access-7swqx\") pod \"e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk\" (UID: \"c08cde99-b619-487f-b34f-3180ba5019be\") " pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk"
Oct 30 19:35:11.818444 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.818357    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e66fa4-rhxq9"]
Oct 30 19:35:11.846648 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.846619    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e66fa4-rhxq9"
Oct 30 19:35:11.872637 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.872590    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d660f20-4a3c-42e1-a518-bd74fa09a673-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-20e66fa4-rhxq9\" (UID: \"7d660f20-4a3c-42e1-a518-bd74fa09a673\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e66fa4-rhxq9"
Oct 30 19:35:11.873227 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.872984    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d660f20-4a3c-42e1-a518-bd74fa09a673-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-20e66fa4-rhxq9\" (UID: \"7d660f20-4a3c-42e1-a518-bd74fa09a673\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e66fa4-rhxq9"
Oct 30 19:35:11.873227 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.873064    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-949qs\" (UniqueName: \"kubernetes.io/projected/7d660f20-4a3c-42e1-a518-bd74fa09a673-kube-api-access-949qs\") pod \"maas-api-revoke-keys-e2e-mt-20e66fa4-rhxq9\" (UID: \"7d660f20-4a3c-42e1-a518-bd74fa09a673\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e66fa4-rhxq9"
Oct 30 19:35:11.875987 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.875083    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e66fa4-rhxq9"]
Oct 30 19:35:11.875987 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.875549    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:35:11.875987 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.875606    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:35:11.875987 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.875688    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:35:11.875987 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.875727    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:35:11.912004 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.911965    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk"
Oct 30 19:35:11.979426 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.977721    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d660f20-4a3c-42e1-a518-bd74fa09a673-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-20e66fa4-rhxq9\" (UID: \"7d660f20-4a3c-42e1-a518-bd74fa09a673\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e66fa4-rhxq9"
Oct 30 19:35:11.979426 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.977817    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d660f20-4a3c-42e1-a518-bd74fa09a673-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-20e66fa4-rhxq9\" (UID: \"7d660f20-4a3c-42e1-a518-bd74fa09a673\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e66fa4-rhxq9"
Oct 30 19:35:11.979426 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.977881    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-949qs\" (UniqueName: \"kubernetes.io/projected/7d660f20-4a3c-42e1-a518-bd74fa09a673-kube-api-access-949qs\") pod \"maas-api-revoke-keys-e2e-mt-20e66fa4-rhxq9\" (UID: \"7d660f20-4a3c-42e1-a518-bd74fa09a673\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e66fa4-rhxq9"
Oct 30 19:35:11.979426 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.979376    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d660f20-4a3c-42e1-a518-bd74fa09a673-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-20e66fa4-rhxq9\" (UID: \"7d660f20-4a3c-42e1-a518-bd74fa09a673\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e66fa4-rhxq9"
Oct 30 19:35:11.987334 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.986433    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d660f20-4a3c-42e1-a518-bd74fa09a673-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-20e66fa4-rhxq9\" (UID: \"7d660f20-4a3c-42e1-a518-bd74fa09a673\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e66fa4-rhxq9"
Oct 30 19:35:11.992302 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:11.990174    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-949qs\" (UniqueName: \"kubernetes.io/projected/7d660f20-4a3c-42e1-a518-bd74fa09a673-kube-api-access-949qs\") pod \"maas-api-revoke-keys-e2e-mt-20e66fa4-rhxq9\" (UID: \"7d660f20-4a3c-42e1-a518-bd74fa09a673\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e66fa4-rhxq9"
Oct 30 19:35:12.196696 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:12.196567    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e66fa4-rhxq9"
Oct 30 19:35:12.301311 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:12.300946    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk"]
Oct 30 19:35:12.305709 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:12.305175    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:35:12.305709 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:12.305264    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:35:12.305709 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:12.305309    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:35:12.511409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:12.511369    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e66fa4-rhxq9"]
Oct 30 19:35:12.511571 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:35:12.511543    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d660f20_4a3c_42e1_a518_bd74fa09a673.slice/crio-f0b96e40aa557b67a70767ba19a938d82de4ce46a6bdf5feb9128b753a0f9f19 WatchSource:0}: Error finding container f0b96e40aa557b67a70767ba19a938d82de4ce46a6bdf5feb9128b753a0f9f19: Status 404 returned error can't find the container with id f0b96e40aa557b67a70767ba19a938d82de4ce46a6bdf5feb9128b753a0f9f19
Oct 30 19:35:12.546415 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:12.546380    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk" event={"ID":"c08cde99-b619-487f-b34f-3180ba5019be","Type":"ContainerStarted","Data":"a21e899d404b92394a64820357f8a6e4b28ee883343ff5bd273ae1759465914b"}
Oct 30 19:35:12.546578 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:12.546445    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk" event={"ID":"c08cde99-b619-487f-b34f-3180ba5019be","Type":"ContainerStarted","Data":"e7d27ede1afb947ab260196cc8c5b176bf120fe76b394342f56a314a055518bb"}
Oct 30 19:35:12.548009 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:12.547986    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e66fa4-rhxq9" event={"ID":"7d660f20-4a3c-42e1-a518-bd74fa09a673","Type":"ContainerStarted","Data":"f0b96e40aa557b67a70767ba19a938d82de4ce46a6bdf5feb9128b753a0f9f19"}
Oct 30 19:35:12.588450 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:12.588348    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk" podStartSLOduration=1.588327031 podStartE2EDuration="1.588327031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:12.57885738 +0000 UTC m=+1132.089917349" watchObservedRunningTime="2026-10-30 19:35:12.588327031 +0000 UTC m=+1132.099387000"
Oct 30 19:35:12.913034 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:12.913006    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk"
Oct 30 19:35:13.042983 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:13.042879    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1383da43-dlbh5"]
Oct 30 19:35:13.048121 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:13.048093    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1383da43-dlbh5"]
Oct 30 19:35:13.564987 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:13.564954    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e66fa4-rhxq9" event={"ID":"7d660f20-4a3c-42e1-a518-bd74fa09a673","Type":"ContainerStarted","Data":"03f6dc2bdc8eaa5ea83434fe26a50d3c3a4f9ac864823302368fb9f61325aea1"}
Oct 30 19:35:13.619607 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:13.608940    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e66fa4-rhxq9" podStartSLOduration=2.608921294 podStartE2EDuration="2.608921294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:13.608353649 +0000 UTC m=+1133.119413617" watchObservedRunningTime="2026-10-30 19:35:13.608921294 +0000 UTC m=+1133.119981264"
Oct 30 19:35:13.920078 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:13.919986    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk"
Oct 30 19:35:14.581208 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:14.581175    5043 generic.go:356] "Generic (PLEG): container finished" podID="7d660f20-4a3c-42e1-a518-bd74fa09a673" containerID="03f6dc2bdc8eaa5ea83434fe26a50d3c3a4f9ac864823302368fb9f61325aea1" exitCode=0
Oct 30 19:35:14.581726 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:14.581245    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e66fa4-rhxq9" event={"ID":"7d660f20-4a3c-42e1-a518-bd74fa09a673","Type":"ContainerDied","Data":"03f6dc2bdc8eaa5ea83434fe26a50d3c3a4f9ac864823302368fb9f61325aea1"}
Oct 30 19:35:14.581836 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:14.581773    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk"
Oct 30 19:35:14.583012 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:14.582988    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk"
Oct 30 19:35:14.692842 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:14.689709    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e70ab44b-6364-4eca-8fb3-c0598ed37cf0" path="/var/lib/kubelet/pods/e70ab44b-6364-4eca-8fb3-c0598ed37cf0/volumes"
Oct 30 19:35:15.824947 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:15.824920    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e66fa4-rhxq9"
Oct 30 19:35:15.902108 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:15.902021    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d660f20-4a3c-42e1-a518-bd74fa09a673-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d660f20-4a3c-42e1-a518-bd74fa09a673-maas-api-xks-ca\") pod \"7d660f20-4a3c-42e1-a518-bd74fa09a673\" (UID: \"7d660f20-4a3c-42e1-a518-bd74fa09a673\") "
Oct 30 19:35:15.902108 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:15.902075    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d660f20-4a3c-42e1-a518-bd74fa09a673-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d660f20-4a3c-42e1-a518-bd74fa09a673-maas-api-openshift-ca\") pod \"7d660f20-4a3c-42e1-a518-bd74fa09a673\" (UID: \"7d660f20-4a3c-42e1-a518-bd74fa09a673\") "
Oct 30 19:35:15.902357 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:15.902152    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d660f20-4a3c-42e1-a518-bd74fa09a673-kube-api-access-949qs\" (UniqueName: \"kubernetes.io/projected/7d660f20-4a3c-42e1-a518-bd74fa09a673-kube-api-access-949qs\") pod \"7d660f20-4a3c-42e1-a518-bd74fa09a673\" (UID: \"7d660f20-4a3c-42e1-a518-bd74fa09a673\") "
Oct 30 19:35:15.902508 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:15.902470    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d660f20-4a3c-42e1-a518-bd74fa09a673-maas-api-openshift-ca" pod "7d660f20-4a3c-42e1-a518-bd74fa09a673" (UID: "7d660f20-4a3c-42e1-a518-bd74fa09a673"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:15.903104 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:15.903083    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d660f20-4a3c-42e1-a518-bd74fa09a673-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:15.905211 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:15.905186    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d660f20-4a3c-42e1-a518-bd74fa09a673-kube-api-access-949qs" pod "7d660f20-4a3c-42e1-a518-bd74fa09a673" (UID: "7d660f20-4a3c-42e1-a518-bd74fa09a673"). InnerVolumeSpecName "kube-api-access-949qs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:15.905323 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:15.905257    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d660f20-4a3c-42e1-a518-bd74fa09a673-maas-api-xks-ca" pod "7d660f20-4a3c-42e1-a518-bd74fa09a673" (UID: "7d660f20-4a3c-42e1-a518-bd74fa09a673"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:35:16.004718 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:16.004685    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d660f20-4a3c-42e1-a518-bd74fa09a673-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:16.004718 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:16.004716    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-949qs\" (UniqueName: \"kubernetes.io/projected/7d660f20-4a3c-42e1-a518-bd74fa09a673-kube-api-access-949qs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:16.213303 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:16.213274    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-95d2ac-75f686bc4d-v7vw5"]
Oct 30 19:35:16.214178 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:16.214161    5043 cpu_manager.go:414] "containerMap: removing container" podUID="7d660f20-4a3c-42e1-a518-bd74fa09a673" containerName="revoke-keys"
Oct 30 19:35:16.214244 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:16.214183    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d660f20-4a3c-42e1-a518-bd74fa09a673" containerName="revoke-keys"
Oct 30 19:35:16.214375 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:16.214366    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d660f20-4a3c-42e1-a518-bd74fa09a673" containerName="revoke-keys"
Oct 30 19:35:16.218036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:16.218020    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-95d2ac-75f686bc4d-v7vw5"
Oct 30 19:35:16.225502 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:16.225464    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-95d2ac\""
Oct 30 19:35:16.230412 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:16.230382    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-95d2ac-75f686bc4d-v7vw5"]
Oct 30 19:35:16.261395 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:16.261353    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-95d2ac-74dbfdbb99-fjbtf"]
Oct 30 19:35:16.268464 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:16.268435    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-95d2ac-74dbfdbb99-fjbtf"
Oct 30 19:35:16.273898 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:16.273871    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-95d2ac-dockercfg-bx6vx\""
Oct 30 19:35:16.280599 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:16.280561    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-95d2ac-74dbfdbb99-fjbtf"]
Oct 30 19:35:16.312115 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:16.312071    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e5c13a6-0354-4d98-991a-ceb6ea271fb1-plugins-config-volume\") pod \"payload-processing-e2e-api-b-95d2ac-75f686bc4d-v7vw5\" (UID: \"6e5c13a6-0354-4d98-991a-ceb6ea271fb1\") " pod="openshift-ingress/payload-processing-e2e-api-b-95d2ac-75f686bc4d-v7vw5"
Oct 30 19:35:16.312314 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:16.312269    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfhr6\" (UniqueName: \"kubernetes.io/projected/6e5c13a6-0354-4d98-991a-ceb6ea271fb1-kube-api-access-jfhr6\") pod \"payload-processing-e2e-api-b-95d2ac-75f686bc4d-v7vw5\" (UID: \"6e5c13a6-0354-4d98-991a-ceb6ea271fb1\") " pod="openshift-ingress/payload-processing-e2e-api-b-95d2ac-75f686bc4d-v7vw5"
Oct 30 19:35:16.422182 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:16.422147    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e5c13a6-0354-4d98-991a-ceb6ea271fb1-plugins-config-volume\") pod \"payload-processing-e2e-api-b-95d2ac-75f686bc4d-v7vw5\" (UID: \"6e5c13a6-0354-4d98-991a-ceb6ea271fb1\") " pod="openshift-ingress/payload-processing-e2e-api-b-95d2ac-75f686bc4d-v7vw5"
Oct 30 19:35:16.659139 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:16.659047    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e5c13a6-0354-4d98-991a-ceb6ea271fb1-plugins-config-volume\") pod \"payload-processing-e2e-api-b-95d2ac-75f686bc4d-v7vw5\" (UID: \"6e5c13a6-0354-4d98-991a-ceb6ea271fb1\") " pod="openshift-ingress/payload-processing-e2e-api-b-95d2ac-75f686bc4d-v7vw5"
Oct 30 19:35:16.659393 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:16.659369    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgn92\" (UniqueName: \"kubernetes.io/projected/2133d6b7-b08a-46ac-b888-2248e3d9e981-kube-api-access-pgn92\") pod \"payload-pre-processing-e2e-api-b-95d2ac-74dbfdbb99-fjbtf\" (UID: \"2133d6b7-b08a-46ac-b888-2248e3d9e981\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-95d2ac-74dbfdbb99-fjbtf"
Oct 30 19:35:16.659772 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:16.659720    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfhr6\" (UniqueName: \"kubernetes.io/projected/6e5c13a6-0354-4d98-991a-ceb6ea271fb1-kube-api-access-jfhr6\") pod \"payload-processing-e2e-api-b-95d2ac-75f686bc4d-v7vw5\" (UID: \"6e5c13a6-0354-4d98-991a-ceb6ea271fb1\") " pod="openshift-ingress/payload-processing-e2e-api-b-95d2ac-75f686bc4d-v7vw5"
Oct 30 19:35:16.659985 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:16.659967    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2133d6b7-b08a-46ac-b888-2248e3d9e981-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-95d2ac-74dbfdbb99-fjbtf\" (UID: \"2133d6b7-b08a-46ac-b888-2248e3d9e981\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-95d2ac-74dbfdbb99-fjbtf"
Oct 30 19:35:16.740801 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:16.740710    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfhr6\" (UniqueName: \"kubernetes.io/projected/6e5c13a6-0354-4d98-991a-ceb6ea271fb1-kube-api-access-jfhr6\") pod \"payload-processing-e2e-api-b-95d2ac-75f686bc4d-v7vw5\" (UID: \"6e5c13a6-0354-4d98-991a-ceb6ea271fb1\") " pod="openshift-ingress/payload-processing-e2e-api-b-95d2ac-75f686bc4d-v7vw5"
Oct 30 19:35:16.804762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:16.779526    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgn92\" (UniqueName: \"kubernetes.io/projected/2133d6b7-b08a-46ac-b888-2248e3d9e981-kube-api-access-pgn92\") pod \"payload-pre-processing-e2e-api-b-95d2ac-74dbfdbb99-fjbtf\" (UID: \"2133d6b7-b08a-46ac-b888-2248e3d9e981\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-95d2ac-74dbfdbb99-fjbtf"
Oct 30 19:35:16.806427 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:16.806391    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2133d6b7-b08a-46ac-b888-2248e3d9e981-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-95d2ac-74dbfdbb99-fjbtf\" (UID: \"2133d6b7-b08a-46ac-b888-2248e3d9e981\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-95d2ac-74dbfdbb99-fjbtf"
Oct 30 19:35:16.820358 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:16.799148    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgn92\" (UniqueName: \"kubernetes.io/projected/2133d6b7-b08a-46ac-b888-2248e3d9e981-kube-api-access-pgn92\") pod \"payload-pre-processing-e2e-api-b-95d2ac-74dbfdbb99-fjbtf\" (UID: \"2133d6b7-b08a-46ac-b888-2248e3d9e981\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-95d2ac-74dbfdbb99-fjbtf"
Oct 30 19:35:16.820606 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:16.807475    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2133d6b7-b08a-46ac-b888-2248e3d9e981-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-95d2ac-74dbfdbb99-fjbtf\" (UID: \"2133d6b7-b08a-46ac-b888-2248e3d9e981\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-95d2ac-74dbfdbb99-fjbtf"
Oct 30 19:35:16.840166 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:16.840128    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-95d2ac-75f686bc4d-v7vw5"
Oct 30 19:35:16.873921 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:16.873896    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e66fa4-rhxq9"
Oct 30 19:35:16.876515 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:16.876461    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e66fa4-rhxq9" event={"ID":"7d660f20-4a3c-42e1-a518-bd74fa09a673","Type":"ContainerDied","Data":"f0b96e40aa557b67a70767ba19a938d82de4ce46a6bdf5feb9128b753a0f9f19"}
Oct 30 19:35:16.876707 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:16.876521    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0b96e40aa557b67a70767ba19a938d82de4ce46a6bdf5feb9128b753a0f9f19"
Oct 30 19:35:16.894674 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:16.894429    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-95d2ac-74dbfdbb99-fjbtf"
Oct 30 19:35:17.272092 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:35:17.272052    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2133d6b7_b08a_46ac_b888_2248e3d9e981.slice/crio-26279f832a8f0d05635c7ba1ca4aac29a96ad7788eae73386feb9302aebaa975 WatchSource:0}: Error finding container 26279f832a8f0d05635c7ba1ca4aac29a96ad7788eae73386feb9302aebaa975: Status 404 returned error can't find the container with id 26279f832a8f0d05635c7ba1ca4aac29a96ad7788eae73386feb9302aebaa975
Oct 30 19:35:17.273522 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:17.273500    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-95d2ac-74dbfdbb99-fjbtf"]
Oct 30 19:35:17.288104 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:17.288072    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-95d2ac-75f686bc4d-v7vw5"]
Oct 30 19:35:17.888414 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:17.888369    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-95d2ac-74dbfdbb99-fjbtf" event={"ID":"2133d6b7-b08a-46ac-b888-2248e3d9e981","Type":"ContainerStarted","Data":"96952b99923f24adae5e9bce715087dafd62bee07d71fa923cba2aafdc421eb1"}
Oct 30 19:35:17.888955 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:17.888440    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-95d2ac-74dbfdbb99-fjbtf" event={"ID":"2133d6b7-b08a-46ac-b888-2248e3d9e981","Type":"ContainerStarted","Data":"26279f832a8f0d05635c7ba1ca4aac29a96ad7788eae73386feb9302aebaa975"}
Oct 30 19:35:17.888955 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:17.888473    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-95d2ac-74dbfdbb99-fjbtf"
Oct 30 19:35:17.891107 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:17.891076    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-95d2ac-75f686bc4d-v7vw5" event={"ID":"6e5c13a6-0354-4d98-991a-ceb6ea271fb1","Type":"ContainerStarted","Data":"fbd6b833037b34de08ecd362089deac03d9e2ae8b28ee1b5484d849860bb8b7a"}
Oct 30 19:35:17.891277 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:17.891135    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-95d2ac-75f686bc4d-v7vw5" event={"ID":"6e5c13a6-0354-4d98-991a-ceb6ea271fb1","Type":"ContainerStarted","Data":"50ba9a4e3f9991d3004df87c466e99e2b7e2df2c9d90e4b328259ab93d7c31ea"}
Oct 30 19:35:17.891277 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:17.891165    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-95d2ac-75f686bc4d-v7vw5"
Oct 30 19:35:17.906787 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:17.906725    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-95d2ac-74dbfdbb99-fjbtf" podStartSLOduration=1.90671184 podStartE2EDuration="1.90671184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:17.904890555 +0000 UTC m=+1137.415950520" watchObservedRunningTime="2026-10-30 19:35:17.90671184 +0000 UTC m=+1137.417771806"
Oct 30 19:35:17.925070 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:17.925020    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-95d2ac-75f686bc4d-v7vw5" podStartSLOduration=1.925003304 podStartE2EDuration="1.925003304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:17.922281847 +0000 UTC m=+1137.433341821" watchObservedRunningTime="2026-10-30 19:35:17.925003304 +0000 UTC m=+1137.436063266"
Oct 30 19:35:18.030022 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:18.029971    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec322b7e-4rvf8"]
Oct 30 19:35:18.041669 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:18.041628    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec322b7e-4rvf8"]
Oct 30 19:35:18.648363 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:18.648335    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e5a119b-b436-41dd-9d61-70095e33bb6f" path="/var/lib/kubelet/pods/7e5a119b-b436-41dd-9d61-70095e33bb6f/volumes"
Oct 30 19:35:21.585700 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:21.585663    5043 scope.go:122] "RemoveContainer" containerID="d5a29266789cbf17a2ef17f9bbda4f67a58d1503efb480565719e49c151a5de0"
Oct 30 19:35:21.598487 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:21.598460    5043 scope.go:122] "RemoveContainer" containerID="6599f50ffe00618be78a309b8e72facb237131b2aae0c17a192fa97afebe66ec"
Oct 30 19:35:21.610907 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:21.610883    5043 scope.go:122] "RemoveContainer" containerID="0de2733df249e7c88fe7602be83e4e4ba4dd3bc076639286d4019cb610e731a0"
Oct 30 19:35:21.626365 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:21.626342    5043 scope.go:122] "RemoveContainer" containerID="c8cfefb833025748cd52190194964a2bbf0a0efb0f79bd5243f55381c95e4c72"
Oct 30 19:35:21.637839 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:21.637708    5043 scope.go:122] "RemoveContainer" containerID="fa58e3f8dcfdb51a943f3ff23caeb05df56f525abc025ed8e4bd1a88e2fc06c0"
Oct 30 19:35:21.651312 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:21.651285    5043 scope.go:122] "RemoveContainer" containerID="f41e7fe084339020c51b51b0770b499bf29fcffa24e91e1c2acd320e22fb6e3c"
Oct 30 19:35:21.942855 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:21.942767    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-j8gh8"]
Oct 30 19:35:21.956422 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:21.956383    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-7v4rf"]
Oct 30 19:35:24.979961 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:35:24.979919    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:35:26.943533 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:26.943473    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-j8gh8" podUID="98147a81-fd6b-423f-9a9c-2765c284d622" containerName="payload-processing" containerID="cri-o://01c64c2d7ac81f20c0f671f72d848316769780b4a012a838bf751d3a1ae60833" gracePeriod=25
Oct 30 19:35:26.956813 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:26.956776    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-7v4rf" podUID="143d86ee-f169-466e-8232-e4669c5fe647" containerName="payload-pre-processing" containerID="cri-o://d7ac839008133779d04e18a62e828d0c8017363763622557588e97bdf79d54a4" gracePeriod=25
Oct 30 19:35:28.906248 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:28.906216    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-95d2ac-74dbfdbb99-fjbtf"
Oct 30 19:35:28.906642 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:28.906383    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-95d2ac-75f686bc4d-v7vw5"
Oct 30 19:35:29.701837 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:29.701781    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-j8gh8" podUID="98147a81-fd6b-423f-9a9c-2765c284d622" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:35:29.718667 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:29.718624    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-7v4rf" podUID="143d86ee-f169-466e-8232-e4669c5fe647" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:35:32.269123 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:32.269081    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74211168-bplxw"]
Oct 30 19:35:32.277476 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:32.276709    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74211168-bplxw"
Oct 30 19:35:32.280956 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:32.280929    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:35:32.281546 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:32.281522    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74211168-bplxw"]
Oct 30 19:35:32.281870 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:32.281847    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:35:32.282043 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:32.282025    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:35:32.282128 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:32.282094    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:35:32.358232 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:32.358184    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd788ac5-1205-49d2-8ed4-8c22703a79ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74211168-bplxw\" (UID: \"fd788ac5-1205-49d2-8ed4-8c22703a79ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74211168-bplxw"
Oct 30 19:35:32.358405 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:32.358244    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd788ac5-1205-49d2-8ed4-8c22703a79ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74211168-bplxw\" (UID: \"fd788ac5-1205-49d2-8ed4-8c22703a79ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74211168-bplxw"
Oct 30 19:35:32.358725 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:32.358691    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjrp7\" (UniqueName: \"kubernetes.io/projected/fd788ac5-1205-49d2-8ed4-8c22703a79ec-kube-api-access-mjrp7\") pod \"maas-api-revoke-sub-74211168-bplxw\" (UID: \"fd788ac5-1205-49d2-8ed4-8c22703a79ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74211168-bplxw"
Oct 30 19:35:32.461113 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:32.461075    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd788ac5-1205-49d2-8ed4-8c22703a79ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74211168-bplxw\" (UID: \"fd788ac5-1205-49d2-8ed4-8c22703a79ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74211168-bplxw"
Oct 30 19:35:32.461321 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:32.461133    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd788ac5-1205-49d2-8ed4-8c22703a79ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74211168-bplxw\" (UID: \"fd788ac5-1205-49d2-8ed4-8c22703a79ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74211168-bplxw"
Oct 30 19:35:32.461568 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:32.461540    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjrp7\" (UniqueName: \"kubernetes.io/projected/fd788ac5-1205-49d2-8ed4-8c22703a79ec-kube-api-access-mjrp7\") pod \"maas-api-revoke-sub-74211168-bplxw\" (UID: \"fd788ac5-1205-49d2-8ed4-8c22703a79ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74211168-bplxw"
Oct 30 19:35:32.462063 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:32.462036    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd788ac5-1205-49d2-8ed4-8c22703a79ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74211168-bplxw\" (UID: \"fd788ac5-1205-49d2-8ed4-8c22703a79ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74211168-bplxw"
Oct 30 19:35:32.464121 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:32.464096    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd788ac5-1205-49d2-8ed4-8c22703a79ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74211168-bplxw\" (UID: \"fd788ac5-1205-49d2-8ed4-8c22703a79ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74211168-bplxw"
Oct 30 19:35:32.470728 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:32.470701    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjrp7\" (UniqueName: \"kubernetes.io/projected/fd788ac5-1205-49d2-8ed4-8c22703a79ec-kube-api-access-mjrp7\") pod \"maas-api-revoke-sub-74211168-bplxw\" (UID: \"fd788ac5-1205-49d2-8ed4-8c22703a79ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74211168-bplxw"
Oct 30 19:35:32.604552 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:32.604457    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74211168-bplxw"
Oct 30 19:35:33.017101 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:35:33.017067    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd788ac5_1205_49d2_8ed4_8c22703a79ec.slice/crio-c54f4ef4632fa5747cb0fed39332252eea865aa0aee7278c6070b62dfedf8ab2 WatchSource:0}: Error finding container c54f4ef4632fa5747cb0fed39332252eea865aa0aee7278c6070b62dfedf8ab2: Status 404 returned error can't find the container with id c54f4ef4632fa5747cb0fed39332252eea865aa0aee7278c6070b62dfedf8ab2
Oct 30 19:35:33.019864 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:33.019831    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74211168-bplxw"]
Oct 30 19:35:33.242616 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:33.242578    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74211168-bplxw" event={"ID":"fd788ac5-1205-49d2-8ed4-8c22703a79ec","Type":"ContainerStarted","Data":"c54f4ef4632fa5747cb0fed39332252eea865aa0aee7278c6070b62dfedf8ab2"}
Oct 30 19:35:34.047395 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:34.047351    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cc1938a-bblth"]
Oct 30 19:35:34.052646 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:34.052617    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cc1938a-bblth"]
Oct 30 19:35:34.258406 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:34.258355    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74211168-bplxw" event={"ID":"fd788ac5-1205-49d2-8ed4-8c22703a79ec","Type":"ContainerStarted","Data":"ea5c03c777d8a153bfb8f787cfd04b1610ed1c2c1a4c658e83f1ebbdeddc3751"}
Oct 30 19:35:34.278900 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:34.278842    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74211168-bplxw" podStartSLOduration=2.278823652 podStartE2EDuration="2.278823652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:34.275943759 +0000 UTC m=+1153.787003725" watchObservedRunningTime="2026-10-30 19:35:34.278823652 +0000 UTC m=+1153.789883618"
Oct 30 19:35:34.648264 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:34.648185    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8451c454-50d0-45ef-b0c8-7b81f855e6db" path="/var/lib/kubelet/pods/8451c454-50d0-45ef-b0c8-7b81f855e6db/volumes"
Oct 30 19:35:35.272145 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:35.272110    5043 generic.go:356] "Generic (PLEG): container finished" podID="fd788ac5-1205-49d2-8ed4-8c22703a79ec" containerID="ea5c03c777d8a153bfb8f787cfd04b1610ed1c2c1a4c658e83f1ebbdeddc3751" exitCode=0
Oct 30 19:35:35.272556 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:35.272196    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74211168-bplxw" event={"ID":"fd788ac5-1205-49d2-8ed4-8c22703a79ec","Type":"ContainerDied","Data":"ea5c03c777d8a153bfb8f787cfd04b1610ed1c2c1a4c658e83f1ebbdeddc3751"}
Oct 30 19:35:36.576866 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:36.573918    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74211168-bplxw"
Oct 30 19:35:36.626030 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:36.625952    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd788ac5-1205-49d2-8ed4-8c22703a79ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd788ac5-1205-49d2-8ed4-8c22703a79ec-maas-api-openshift-ca\") pod \"fd788ac5-1205-49d2-8ed4-8c22703a79ec\" (UID: \"fd788ac5-1205-49d2-8ed4-8c22703a79ec\") "
Oct 30 19:35:36.626203 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:36.626125    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd788ac5-1205-49d2-8ed4-8c22703a79ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd788ac5-1205-49d2-8ed4-8c22703a79ec-maas-api-xks-ca\") pod \"fd788ac5-1205-49d2-8ed4-8c22703a79ec\" (UID: \"fd788ac5-1205-49d2-8ed4-8c22703a79ec\") "
Oct 30 19:35:36.626249 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:36.626226    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd788ac5-1205-49d2-8ed4-8c22703a79ec-kube-api-access-mjrp7\" (UniqueName: \"kubernetes.io/projected/fd788ac5-1205-49d2-8ed4-8c22703a79ec-kube-api-access-mjrp7\") pod \"fd788ac5-1205-49d2-8ed4-8c22703a79ec\" (UID: \"fd788ac5-1205-49d2-8ed4-8c22703a79ec\") "
Oct 30 19:35:36.626719 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:36.626681    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd788ac5-1205-49d2-8ed4-8c22703a79ec-maas-api-openshift-ca" pod "fd788ac5-1205-49d2-8ed4-8c22703a79ec" (UID: "fd788ac5-1205-49d2-8ed4-8c22703a79ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:36.627128 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:36.627100    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd788ac5-1205-49d2-8ed4-8c22703a79ec-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:36.633698 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:36.631891    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd788ac5-1205-49d2-8ed4-8c22703a79ec-maas-api-xks-ca" pod "fd788ac5-1205-49d2-8ed4-8c22703a79ec" (UID: "fd788ac5-1205-49d2-8ed4-8c22703a79ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:35:36.633698 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:36.631930    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd788ac5-1205-49d2-8ed4-8c22703a79ec-kube-api-access-mjrp7" pod "fd788ac5-1205-49d2-8ed4-8c22703a79ec" (UID: "fd788ac5-1205-49d2-8ed4-8c22703a79ec"). InnerVolumeSpecName "kube-api-access-mjrp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:36.728603 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:36.728569    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd788ac5-1205-49d2-8ed4-8c22703a79ec-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:36.728603 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:36.728605    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjrp7\" (UniqueName: \"kubernetes.io/projected/fd788ac5-1205-49d2-8ed4-8c22703a79ec-kube-api-access-mjrp7\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:37.311764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:37.311236    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74211168-bplxw" event={"ID":"fd788ac5-1205-49d2-8ed4-8c22703a79ec","Type":"ContainerDied","Data":"c54f4ef4632fa5747cb0fed39332252eea865aa0aee7278c6070b62dfedf8ab2"}
Oct 30 19:35:37.311764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:37.311303    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c54f4ef4632fa5747cb0fed39332252eea865aa0aee7278c6070b62dfedf8ab2"
Oct 30 19:35:37.311764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:37.311373    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74211168-bplxw"
Oct 30 19:35:39.744762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:39.743864    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-j8gh8" podUID="98147a81-fd6b-423f-9a9c-2765c284d622" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:35:39.766781 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:39.745876    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-7v4rf" podUID="143d86ee-f169-466e-8232-e4669c5fe647" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:35:43.031070 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:43.031035    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-235d91c2-bnzkl"]
Oct 30 19:35:43.041904 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:43.041866    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-235d91c2-bnzkl"]
Oct 30 19:35:44.674357 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:44.674325    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="277eda52-9140-4bc2-aacd-4b383b4ca7c5" path="/var/lib/kubelet/pods/277eda52-9140-4bc2-aacd-4b383b4ca7c5/volumes"
Oct 30 19:35:45.036992 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:45.036960    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dff807b2-dtrbk"]
Oct 30 19:35:45.041986 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:45.041961    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dff807b2-dtrbk"]
Oct 30 19:35:46.079856 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:46.079822    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-73f96d-65c48fbd9f-98p49"]
Oct 30 19:35:46.081208 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:46.081183    5043 cpu_manager.go:414] "containerMap: removing container" podUID="fd788ac5-1205-49d2-8ed4-8c22703a79ec" containerName="revoke-keys"
Oct 30 19:35:46.081208 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:46.081204    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd788ac5-1205-49d2-8ed4-8c22703a79ec" containerName="revoke-keys"
Oct 30 19:35:46.081441 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:46.081430    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd788ac5-1205-49d2-8ed4-8c22703a79ec" containerName="revoke-keys"
Oct 30 19:35:46.086856 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:46.086831    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-73f96d-65c48fbd9f-98p49"
Oct 30 19:35:46.093062 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:46.093037    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-73f96d-dockercfg-r4rtm\""
Oct 30 19:35:46.093206 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:46.093084    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-73f96d\""
Oct 30 19:35:46.094279 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:46.094246    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-73f96d-65c48fbd9f-98p49"]
Oct 30 19:35:46.255767 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:46.255579    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f7e3a16-0761-42c5-9797-85fe755780d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-73f96d-65c48fbd9f-98p49\" (UID: \"6f7e3a16-0761-42c5-9797-85fe755780d8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-73f96d-65c48fbd9f-98p49"
Oct 30 19:35:46.295819 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:46.266178    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn572\" (UniqueName: \"kubernetes.io/projected/6f7e3a16-0761-42c5-9797-85fe755780d8-kube-api-access-nn572\") pod \"payload-pre-processing-e2e-api-a-73f96d-65c48fbd9f-98p49\" (UID: \"6f7e3a16-0761-42c5-9797-85fe755780d8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-73f96d-65c48fbd9f-98p49"
Oct 30 19:35:46.385986 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:46.385845    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f7e3a16-0761-42c5-9797-85fe755780d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-73f96d-65c48fbd9f-98p49\" (UID: \"6f7e3a16-0761-42c5-9797-85fe755780d8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-73f96d-65c48fbd9f-98p49"
Oct 30 19:35:46.402809 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:46.387627    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn572\" (UniqueName: \"kubernetes.io/projected/6f7e3a16-0761-42c5-9797-85fe755780d8-kube-api-access-nn572\") pod \"payload-pre-processing-e2e-api-a-73f96d-65c48fbd9f-98p49\" (UID: \"6f7e3a16-0761-42c5-9797-85fe755780d8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-73f96d-65c48fbd9f-98p49"
Oct 30 19:35:46.403937 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:46.387206    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f7e3a16-0761-42c5-9797-85fe755780d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-73f96d-65c48fbd9f-98p49\" (UID: \"6f7e3a16-0761-42c5-9797-85fe755780d8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-73f96d-65c48fbd9f-98p49"
Oct 30 19:35:46.468565 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:46.461637    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn572\" (UniqueName: \"kubernetes.io/projected/6f7e3a16-0761-42c5-9797-85fe755780d8-kube-api-access-nn572\") pod \"payload-pre-processing-e2e-api-a-73f96d-65c48fbd9f-98p49\" (UID: \"6f7e3a16-0761-42c5-9797-85fe755780d8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-73f96d-65c48fbd9f-98p49"
Oct 30 19:35:46.649051 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:46.648969    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f42c708e-e9b6-4cd9-9cd4-090a55157421" path="/var/lib/kubelet/pods/f42c708e-e9b6-4cd9-9cd4-090a55157421/volumes"
Oct 30 19:35:46.709416 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:46.709375    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-73f96d-65c48fbd9f-98p49"
Oct 30 19:35:46.750006 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:46.749976    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c7880b-779lw"]
Oct 30 19:35:46.755475 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:46.755439    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c7880b-779lw"
Oct 30 19:35:46.761107 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:46.761080    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:35:46.762225 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:46.762202    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:35:46.762454 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:46.762441    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:35:46.762622 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:46.762610    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:35:46.767143 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:46.767117    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c7880b-779lw"]
Oct 30 19:35:46.823241 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:46.822805    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c7880b-779lw\" (UID: \"129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c7880b-779lw"
Oct 30 19:35:46.823241 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:46.822978    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c7880b-779lw\" (UID: \"129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c7880b-779lw"
Oct 30 19:35:46.823241 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:46.823173    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t924l\" (UniqueName: \"kubernetes.io/projected/129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61-kube-api-access-t924l\") pod \"maas-api-revoke-keys-e2e-api-a-c7880b-779lw\" (UID: \"129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c7880b-779lw"
Oct 30 19:35:46.925598 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:46.925447    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t924l\" (UniqueName: \"kubernetes.io/projected/129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61-kube-api-access-t924l\") pod \"maas-api-revoke-keys-e2e-api-a-c7880b-779lw\" (UID: \"129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c7880b-779lw"
Oct 30 19:35:46.925805 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:46.925612    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c7880b-779lw\" (UID: \"129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c7880b-779lw"
Oct 30 19:35:46.925805 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:46.925724    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c7880b-779lw\" (UID: \"129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c7880b-779lw"
Oct 30 19:35:46.926992 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:46.926967    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c7880b-779lw\" (UID: \"129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c7880b-779lw"
Oct 30 19:35:46.928598 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:46.928574    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c7880b-779lw\" (UID: \"129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c7880b-779lw"
Oct 30 19:35:46.939115 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:46.938645    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t924l\" (UniqueName: \"kubernetes.io/projected/129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61-kube-api-access-t924l\") pod \"maas-api-revoke-keys-e2e-api-a-c7880b-779lw\" (UID: \"129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c7880b-779lw"
Oct 30 19:35:47.032554 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:47.032531    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-73f96d-65c48fbd9f-98p49"]
Oct 30 19:35:47.085117 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:35:47.085080    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f7e3a16_0761_42c5_9797_85fe755780d8.slice/crio-8f023eaff116fc0bfe86293ef4c71ab9897c86ca30273ee7eea1486925565fc2 WatchSource:0}: Error finding container 8f023eaff116fc0bfe86293ef4c71ab9897c86ca30273ee7eea1486925565fc2: Status 404 returned error can't find the container with id 8f023eaff116fc0bfe86293ef4c71ab9897c86ca30273ee7eea1486925565fc2
Oct 30 19:35:47.119340 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:47.119301    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c7880b-779lw"
Oct 30 19:35:47.298277 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:47.298245    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-j8gh8"
Oct 30 19:35:47.446788 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:47.445083    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98147a81-fd6b-423f-9a9c-2765c284d622-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98147a81-fd6b-423f-9a9c-2765c284d622-plugins-config-volume\") pod \"98147a81-fd6b-423f-9a9c-2765c284d622\" (UID: \"98147a81-fd6b-423f-9a9c-2765c284d622\") "
Oct 30 19:35:47.446788 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:47.445128    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98147a81-fd6b-423f-9a9c-2765c284d622-kube-api-access-f5ccw\" (UniqueName: \"kubernetes.io/projected/98147a81-fd6b-423f-9a9c-2765c284d622-kube-api-access-f5ccw\") pod \"98147a81-fd6b-423f-9a9c-2765c284d622\" (UID: \"98147a81-fd6b-423f-9a9c-2765c284d622\") "
Oct 30 19:35:47.498097 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:47.453788    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98147a81-fd6b-423f-9a9c-2765c284d622-plugins-config-volume" pod "98147a81-fd6b-423f-9a9c-2765c284d622" (UID: "98147a81-fd6b-423f-9a9c-2765c284d622"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:47.510773 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:47.506101    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98147a81-fd6b-423f-9a9c-2765c284d622-kube-api-access-f5ccw" pod "98147a81-fd6b-423f-9a9c-2765c284d622" (UID: "98147a81-fd6b-423f-9a9c-2765c284d622"). InnerVolumeSpecName "kube-api-access-f5ccw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:47.540879 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:47.540847    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-7v4rf"
Oct 30 19:35:47.548949 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:47.548924    5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98147a81-fd6b-423f-9a9c-2765c284d622-plugins-config-volume\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:47.549136 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:47.549121    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5ccw\" (UniqueName: \"kubernetes.io/projected/98147a81-fd6b-423f-9a9c-2765c284d622-kube-api-access-f5ccw\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:47.634278 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:47.634229    5043 generic.go:356] "Generic (PLEG): container finished" podID="143d86ee-f169-466e-8232-e4669c5fe647" containerID="d7ac839008133779d04e18a62e828d0c8017363763622557588e97bdf79d54a4" exitCode=0
Oct 30 19:35:47.634464 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:47.634299    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-7v4rf"
Oct 30 19:35:47.634464 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:47.634313    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-7v4rf" event={"ID":"143d86ee-f169-466e-8232-e4669c5fe647","Type":"ContainerDied","Data":"d7ac839008133779d04e18a62e828d0c8017363763622557588e97bdf79d54a4"}
Oct 30 19:35:47.634464 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:47.634410    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-7v4rf" event={"ID":"143d86ee-f169-466e-8232-e4669c5fe647","Type":"ContainerDied","Data":"97236159d8359348e41f3d70b0d1814e5b5e8ecdc434897c907d67db6ffdbce9"}
Oct 30 19:35:47.634464 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:47.634444    5043 scope.go:122] "RemoveContainer" containerID="d7ac839008133779d04e18a62e828d0c8017363763622557588e97bdf79d54a4"
Oct 30 19:35:47.636832 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:47.636810    5043 generic.go:356] "Generic (PLEG): container finished" podID="98147a81-fd6b-423f-9a9c-2765c284d622" containerID="01c64c2d7ac81f20c0f671f72d848316769780b4a012a838bf751d3a1ae60833" exitCode=0
Oct 30 19:35:47.636958 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:47.636929    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-j8gh8" event={"ID":"98147a81-fd6b-423f-9a9c-2765c284d622","Type":"ContainerDied","Data":"01c64c2d7ac81f20c0f671f72d848316769780b4a012a838bf751d3a1ae60833"}
Oct 30 19:35:47.637015 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:47.636969    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-j8gh8" event={"ID":"98147a81-fd6b-423f-9a9c-2765c284d622","Type":"ContainerDied","Data":"90d9adf405975e44dd1de1e682cf15f0732f0a5bbc2c3dc46de5c62b674f9cdd"}
Oct 30 19:35:47.637901 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:47.637882    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-j8gh8"
Oct 30 19:35:47.640185 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:47.640155    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-73f96d-65c48fbd9f-98p49" event={"ID":"6f7e3a16-0761-42c5-9797-85fe755780d8","Type":"ContainerStarted","Data":"5903d9e55ac21661da7bcd1afcbfc204b75f016071db86a6cfd09ac53e499a01"}
Oct 30 19:35:47.640383 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:47.640211    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-73f96d-65c48fbd9f-98p49" event={"ID":"6f7e3a16-0761-42c5-9797-85fe755780d8","Type":"ContainerStarted","Data":"8f023eaff116fc0bfe86293ef4c71ab9897c86ca30273ee7eea1486925565fc2"}
Oct 30 19:35:47.640514 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:47.640496    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-73f96d-65c48fbd9f-98p49"
Oct 30 19:35:47.649287 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:47.649262    5043 scope.go:122] "RemoveContainer" containerID="d7ac839008133779d04e18a62e828d0c8017363763622557588e97bdf79d54a4"
Oct 30 19:35:47.649649 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:35:47.649627    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7ac839008133779d04e18a62e828d0c8017363763622557588e97bdf79d54a4\": container with ID starting with d7ac839008133779d04e18a62e828d0c8017363763622557588e97bdf79d54a4 not found: ID does not exist" containerID="d7ac839008133779d04e18a62e828d0c8017363763622557588e97bdf79d54a4"
Oct 30 19:35:47.649793 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:47.649662    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7ac839008133779d04e18a62e828d0c8017363763622557588e97bdf79d54a4"} err="failed to get container status \"d7ac839008133779d04e18a62e828d0c8017363763622557588e97bdf79d54a4\": rpc error: code = NotFound desc = could not find container \"d7ac839008133779d04e18a62e828d0c8017363763622557588e97bdf79d54a4\": container with ID starting with d7ac839008133779d04e18a62e828d0c8017363763622557588e97bdf79d54a4 not found: ID does not exist"
Oct 30 19:35:47.649793 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:47.649684    5043 scope.go:122] "RemoveContainer" containerID="01c64c2d7ac81f20c0f671f72d848316769780b4a012a838bf751d3a1ae60833"
Oct 30 19:35:47.649947 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:47.649811    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/143d86ee-f169-466e-8232-e4669c5fe647-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/143d86ee-f169-466e-8232-e4669c5fe647-plugins-config-volume\") pod \"143d86ee-f169-466e-8232-e4669c5fe647\" (UID: \"143d86ee-f169-466e-8232-e4669c5fe647\") "
Oct 30 19:35:47.650014 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:47.649993    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/143d86ee-f169-466e-8232-e4669c5fe647-kube-api-access-bmmbx\" (UniqueName: \"kubernetes.io/projected/143d86ee-f169-466e-8232-e4669c5fe647-kube-api-access-bmmbx\") pod \"143d86ee-f169-466e-8232-e4669c5fe647\" (UID: \"143d86ee-f169-466e-8232-e4669c5fe647\") "
Oct 30 19:35:47.650238 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:47.650215    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/143d86ee-f169-466e-8232-e4669c5fe647-plugins-config-volume" pod "143d86ee-f169-466e-8232-e4669c5fe647" (UID: "143d86ee-f169-466e-8232-e4669c5fe647"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:47.651130 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:47.650899    5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/143d86ee-f169-466e-8232-e4669c5fe647-plugins-config-volume\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:47.652807 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:47.652719    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/143d86ee-f169-466e-8232-e4669c5fe647-kube-api-access-bmmbx" pod "143d86ee-f169-466e-8232-e4669c5fe647" (UID: "143d86ee-f169-466e-8232-e4669c5fe647"). InnerVolumeSpecName "kube-api-access-bmmbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:47.685097 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:47.685055    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c7880b-779lw"]
Oct 30 19:35:47.687706 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:47.687658    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-73f96d-65c48fbd9f-98p49" podStartSLOduration=1.687643879 podStartE2EDuration="1.687643879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:47.674205663 +0000 UTC m=+1167.185265629" watchObservedRunningTime="2026-10-30 19:35:47.687643879 +0000 UTC m=+1167.198703864"
Oct 30 19:35:47.694815 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:47.694795    5043 scope.go:122] "RemoveContainer" containerID="01c64c2d7ac81f20c0f671f72d848316769780b4a012a838bf751d3a1ae60833"
Oct 30 19:35:47.695508 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:35:47.695442    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01c64c2d7ac81f20c0f671f72d848316769780b4a012a838bf751d3a1ae60833\": container with ID starting with 01c64c2d7ac81f20c0f671f72d848316769780b4a012a838bf751d3a1ae60833 not found: ID does not exist" containerID="01c64c2d7ac81f20c0f671f72d848316769780b4a012a838bf751d3a1ae60833"
Oct 30 19:35:47.695639 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:47.695510    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01c64c2d7ac81f20c0f671f72d848316769780b4a012a838bf751d3a1ae60833"} err="failed to get container status \"01c64c2d7ac81f20c0f671f72d848316769780b4a012a838bf751d3a1ae60833\": rpc error: code = NotFound desc = could not find container \"01c64c2d7ac81f20c0f671f72d848316769780b4a012a838bf751d3a1ae60833\": container with ID starting with 01c64c2d7ac81f20c0f671f72d848316769780b4a012a838bf751d3a1ae60833 not found: ID does not exist"
Oct 30 19:35:47.699169 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:35:47.699141    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod129b9e3e_d6b6_40c3_adcc_ff16bcf3ca61.slice/crio-9d8a34b9cd3af127fb55d32fd9b8d82fa013a9636df733bcdbddf3a4fd84d49d WatchSource:0}: Error finding container 9d8a34b9cd3af127fb55d32fd9b8d82fa013a9636df733bcdbddf3a4fd84d49d: Status 404 returned error can't find the container with id 9d8a34b9cd3af127fb55d32fd9b8d82fa013a9636df733bcdbddf3a4fd84d49d
Oct 30 19:35:47.707490 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:47.707463    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-j8gh8"]
Oct 30 19:35:47.712846 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:47.712824    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-j8gh8"]
Oct 30 19:35:47.753192 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:47.753125    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmmbx\" (UniqueName: \"kubernetes.io/projected/143d86ee-f169-466e-8232-e4669c5fe647-kube-api-access-bmmbx\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:47.968420 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:47.968389    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-7v4rf"]
Oct 30 19:35:47.975269 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:47.975241    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-7v4rf"]
Oct 30 19:35:48.028340 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:48.028262    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80a9dfa1-mn2sk"]
Oct 30 19:35:48.033726 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:48.033702    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80a9dfa1-mn2sk"]
Oct 30 19:35:48.693037 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:48.692997    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="143d86ee-f169-466e-8232-e4669c5fe647" path="/var/lib/kubelet/pods/143d86ee-f169-466e-8232-e4669c5fe647/volumes"
Oct 30 19:35:48.693850 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:48.693828    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6095ad00-f596-438a-96c7-80757d6c8284" path="/var/lib/kubelet/pods/6095ad00-f596-438a-96c7-80757d6c8284/volumes"
Oct 30 19:35:48.694438 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:48.694415    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98147a81-fd6b-423f-9a9c-2765c284d622" path="/var/lib/kubelet/pods/98147a81-fd6b-423f-9a9c-2765c284d622/volumes"
Oct 30 19:35:48.698051 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:48.698026    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c7880b-779lw" event={"ID":"129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61","Type":"ContainerStarted","Data":"4bdc9a97d9d5f4e1494f5a2d3f8b3ddd4fc4987c140a304072d2a67ced30fb13"}
Oct 30 19:35:48.698137 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:48.698073    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c7880b-779lw" event={"ID":"129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61","Type":"ContainerStarted","Data":"9d8a34b9cd3af127fb55d32fd9b8d82fa013a9636df733bcdbddf3a4fd84d49d"}
Oct 30 19:35:48.726028 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:48.720271    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c7880b-779lw" podStartSLOduration=2.720249634 podStartE2EDuration="2.720249634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:48.716096951 +0000 UTC m=+1168.227156916" watchObservedRunningTime="2026-10-30 19:35:48.720249634 +0000 UTC m=+1168.231309600"
Oct 30 19:35:49.713794 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:49.713725    5043 generic.go:356] "Generic (PLEG): container finished" podID="129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61" containerID="4bdc9a97d9d5f4e1494f5a2d3f8b3ddd4fc4987c140a304072d2a67ced30fb13" exitCode=0
Oct 30 19:35:49.714207 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:49.713815    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c7880b-779lw" event={"ID":"129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61","Type":"ContainerDied","Data":"4bdc9a97d9d5f4e1494f5a2d3f8b3ddd4fc4987c140a304072d2a67ced30fb13"}
Oct 30 19:35:50.948280 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:50.948255    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c7880b-779lw"
Oct 30 19:35:51.160904 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:51.160815    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61-maas-api-openshift-ca\") pod \"129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61\" (UID: \"129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61\") "
Oct 30 19:35:51.161058 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:51.160941    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61-kube-api-access-t924l\" (UniqueName: \"kubernetes.io/projected/129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61-kube-api-access-t924l\") pod \"129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61\" (UID: \"129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61\") "
Oct 30 19:35:51.161102 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:51.161058    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61-maas-api-xks-ca\") pod \"129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61\" (UID: \"129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61\") "
Oct 30 19:35:51.161275 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:51.161246    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61-maas-api-openshift-ca" pod "129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61" (UID: "129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:51.161993 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:51.161967    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:51.163344 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:51.163315    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61-kube-api-access-t924l" pod "129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61" (UID: "129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61"). InnerVolumeSpecName "kube-api-access-t924l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:51.163432 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:51.163354    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61-maas-api-xks-ca" pod "129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61" (UID: "129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:35:51.264107 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:51.264070    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t924l\" (UniqueName: \"kubernetes.io/projected/129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61-kube-api-access-t924l\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:51.264107 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:51.264104    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:51.735019 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:51.734990    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c7880b-779lw"
Oct 30 19:35:51.735225 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:51.735033    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c7880b-779lw" event={"ID":"129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61","Type":"ContainerDied","Data":"9d8a34b9cd3af127fb55d32fd9b8d82fa013a9636df733bcdbddf3a4fd84d49d"}
Oct 30 19:35:51.735225 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:51.735084    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d8a34b9cd3af127fb55d32fd9b8d82fa013a9636df733bcdbddf3a4fd84d49d"
Oct 30 19:35:52.031025 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:52.030946    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b25eed1-4dkn6"]
Oct 30 19:35:52.036585 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:52.036562    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b25eed1-4dkn6"]
Oct 30 19:35:52.041409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:52.041385    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5166f1e7-j5jqp"]
Oct 30 19:35:52.046013 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:52.045989    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5166f1e7-j5jqp"]
Oct 30 19:35:52.648541 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:52.648505    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a377cda-465d-45ca-bdd3-5bb94c19ed24" path="/var/lib/kubelet/pods/1a377cda-465d-45ca-bdd3-5bb94c19ed24/volumes"
Oct 30 19:35:52.649095 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:52.649079    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4c85361-fb0b-49e4-8f86-3fb92a60ab61" path="/var/lib/kubelet/pods/d4c85361-fb0b-49e4-8f86-3fb92a60ab61/volumes"
Oct 30 19:35:53.034845 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:53.034813    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c91cb764-knbd8"]
Oct 30 19:35:53.041003 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:53.040973    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c91cb764-knbd8"]
Oct 30 19:35:53.973875 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:53.973840    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e66fa4-rhxq9"]
Oct 30 19:35:53.985251 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:53.983867    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e66fa4-rhxq9"]
Oct 30 19:35:54.649090 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:54.649051    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d660f20-4a3c-42e1-a518-bd74fa09a673" path="/var/lib/kubelet/pods/7d660f20-4a3c-42e1-a518-bd74fa09a673/volumes"
Oct 30 19:35:54.649821 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:54.649799    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1c894c4-4433-4736-b57e-b6bf5db5e698" path="/var/lib/kubelet/pods/a1c894c4-4433-4736-b57e-b6bf5db5e698/volumes"
Oct 30 19:35:54.981673 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:35:54.981631    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:35:56.888159 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:56.888125    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c7880b-d5b8b5f94-gwr52"]
Oct 30 19:35:56.910502 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:56.910471    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c7880b-c46dcb8b7-dkz2k"]
Oct 30 19:35:57.813945 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:57.813906    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv"]
Oct 30 19:35:57.821568 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:57.821502    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv" podUID="23f8f6fc-85c3-46ac-88f5-08e08f246560" containerName="istio-proxy" containerID="cri-o://8b068dfe8707cc390ec583f8fa4bc639edaf1fdabf87227a1d758ab4b49d149e" gracePeriod=30
Oct 30 19:35:57.827586 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:57.827550    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq"]
Oct 30 19:35:57.827942 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:57.827883    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq" podUID="212ce985-524a-4391-8926-63aafdc58132" containerName="istio-proxy" containerID="cri-o://35ffcde418b19f88d351e011ed9b566bfe3f6111991ee359720fd4947da07efb" gracePeriod=30
Oct 30 19:35:58.704242 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:35:58.704216    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-73f96d-65c48fbd9f-98p49"
Oct 30 19:36:01.033728 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:01.033693    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0877e187-h82kf"]
Oct 30 19:36:01.038922 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:01.038895    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0877e187-h82kf"]
Oct 30 19:36:01.888612 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:01.888571    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-c7880b-d5b8b5f94-gwr52" podUID="9d198eae-9067-4789-b6b1-b8827ca742ab" containerName="payload-processing" containerID="cri-o://0ca3205e20388759dd42f2e1f7f6acc2fb9f01853c0924c43ed5f69560fd6937" gracePeriod=25
Oct 30 19:36:01.911703 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:01.911663    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c7880b-c46dcb8b7-dkz2k" podUID="b7ee29a5-f392-4c4f-907b-dac7c0140f6c" containerName="payload-pre-processing" containerID="cri-o://1a739e75c29e65878454639cd97bc127d806eb8ccc45087f347ce0f17a32dd95" gracePeriod=25
Oct 30 19:36:02.229659 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.229618    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-hsg4m"]
Oct 30 19:36:02.231412 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.231380    5043 cpu_manager.go:414] "containerMap: removing container" podUID="98147a81-fd6b-423f-9a9c-2765c284d622" containerName="payload-processing"
Oct 30 19:36:02.231412 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.231412    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98147a81-fd6b-423f-9a9c-2765c284d622" containerName="payload-processing"
Oct 30 19:36:02.231610 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.231439    5043 cpu_manager.go:414] "containerMap: removing container" podUID="143d86ee-f169-466e-8232-e4669c5fe647" containerName="payload-pre-processing"
Oct 30 19:36:02.231610 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.231447    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="143d86ee-f169-466e-8232-e4669c5fe647" containerName="payload-pre-processing"
Oct 30 19:36:02.231610 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.231469    5043 cpu_manager.go:414] "containerMap: removing container" podUID="129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61" containerName="revoke-keys"
Oct 30 19:36:02.231610 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.231476    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61" containerName="revoke-keys"
Oct 30 19:36:02.231972 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.231949    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="143d86ee-f169-466e-8232-e4669c5fe647" containerName="payload-pre-processing"
Oct 30 19:36:02.232069 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.231984    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61" containerName="revoke-keys"
Oct 30 19:36:02.232069 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.232025    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="98147a81-fd6b-423f-9a9c-2765c284d622" containerName="payload-processing"
Oct 30 19:36:02.254906 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.254854    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-hsg4m"
Oct 30 19:36:02.289869 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.287648    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-20e66fa4\""
Oct 30 19:36:02.300494 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.300452    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe77a5ed-7a32-4350-b60c-d119d766eb1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-20e66fa4-997965c87-hsg4m\" (UID: \"fe77a5ed-7a32-4350-b60c-d119d766eb1c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-hsg4m"
Oct 30 19:36:02.300792 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.300774    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6dbx\" (UniqueName: \"kubernetes.io/projected/fe77a5ed-7a32-4350-b60c-d119d766eb1c-kube-api-access-z6dbx\") pod \"payload-pre-processing-e2e-mt-20e66fa4-997965c87-hsg4m\" (UID: \"fe77a5ed-7a32-4350-b60c-d119d766eb1c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-hsg4m"
Oct 30 19:36:02.315860 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.304405    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-20e66fa4-dockercfg-2qfx8\""
Oct 30 19:36:02.654514 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.654472    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6dbx\" (UniqueName: \"kubernetes.io/projected/fe77a5ed-7a32-4350-b60c-d119d766eb1c-kube-api-access-z6dbx\") pod \"payload-pre-processing-e2e-mt-20e66fa4-997965c87-hsg4m\" (UID: \"fe77a5ed-7a32-4350-b60c-d119d766eb1c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-hsg4m"
Oct 30 19:36:02.655901 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.655223    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe77a5ed-7a32-4350-b60c-d119d766eb1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-20e66fa4-997965c87-hsg4m\" (UID: \"fe77a5ed-7a32-4350-b60c-d119d766eb1c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-hsg4m"
Oct 30 19:36:02.657180 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.657156    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe77a5ed-7a32-4350-b60c-d119d766eb1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-20e66fa4-997965c87-hsg4m\" (UID: \"fe77a5ed-7a32-4350-b60c-d119d766eb1c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-hsg4m"
Oct 30 19:36:02.658706 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.658678    5043 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:36:02.658842 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.658817    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq" podUID="212ce985-524a-4391-8926-63aafdc58132" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:36:02.671335 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.671304    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c7880b-c46dcb8b7-dkz2k"
Oct 30 19:36:02.681472 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.681436    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c7880b-d5b8b5f94-gwr52"
Oct 30 19:36:02.685932 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.685892    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6dbx\" (UniqueName: \"kubernetes.io/projected/fe77a5ed-7a32-4350-b60c-d119d766eb1c-kube-api-access-z6dbx\") pod \"payload-pre-processing-e2e-mt-20e66fa4-997965c87-hsg4m\" (UID: \"fe77a5ed-7a32-4350-b60c-d119d766eb1c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-hsg4m"
Oct 30 19:36:02.750813 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.730206    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d8f8137-26e5-4a07-8dd4-1155cdd6d896" path="/var/lib/kubelet/pods/7d8f8137-26e5-4a07-8dd4-1155cdd6d896/volumes"
Oct 30 19:36:02.804663 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.769896    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d198eae-9067-4789-b6b1-b8827ca742ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d198eae-9067-4789-b6b1-b8827ca742ab-plugins-config-volume\") pod \"9d198eae-9067-4789-b6b1-b8827ca742ab\" (UID: \"9d198eae-9067-4789-b6b1-b8827ca742ab\") "
Oct 30 19:36:02.804663 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.770119    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7ee29a5-f392-4c4f-907b-dac7c0140f6c-kube-api-access-p65vz\" (UniqueName: \"kubernetes.io/projected/b7ee29a5-f392-4c4f-907b-dac7c0140f6c-kube-api-access-p65vz\") pod \"b7ee29a5-f392-4c4f-907b-dac7c0140f6c\" (UID: \"b7ee29a5-f392-4c4f-907b-dac7c0140f6c\") "
Oct 30 19:36:02.804663 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.770170    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d198eae-9067-4789-b6b1-b8827ca742ab-kube-api-access-r74lq\" (UniqueName: \"kubernetes.io/projected/9d198eae-9067-4789-b6b1-b8827ca742ab-kube-api-access-r74lq\") pod \"9d198eae-9067-4789-b6b1-b8827ca742ab\" (UID: \"9d198eae-9067-4789-b6b1-b8827ca742ab\") "
Oct 30 19:36:02.804663 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.770288    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7ee29a5-f392-4c4f-907b-dac7c0140f6c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7ee29a5-f392-4c4f-907b-dac7c0140f6c-plugins-config-volume\") pod \"b7ee29a5-f392-4c4f-907b-dac7c0140f6c\" (UID: \"b7ee29a5-f392-4c4f-907b-dac7c0140f6c\") "
Oct 30 19:36:02.804663 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.772356    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d198eae-9067-4789-b6b1-b8827ca742ab-plugins-config-volume" pod "9d198eae-9067-4789-b6b1-b8827ca742ab" (UID: "9d198eae-9067-4789-b6b1-b8827ca742ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:36:02.804663 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.782556    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7ee29a5-f392-4c4f-907b-dac7c0140f6c-plugins-config-volume" pod "b7ee29a5-f392-4c4f-907b-dac7c0140f6c" (UID: "b7ee29a5-f392-4c4f-907b-dac7c0140f6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:36:02.806839 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.805451    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d198eae-9067-4789-b6b1-b8827ca742ab-kube-api-access-r74lq" pod "9d198eae-9067-4789-b6b1-b8827ca742ab" (UID: "9d198eae-9067-4789-b6b1-b8827ca742ab"). InnerVolumeSpecName "kube-api-access-r74lq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:02.806839 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.805569    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7ee29a5-f392-4c4f-907b-dac7c0140f6c-kube-api-access-p65vz" pod "b7ee29a5-f392-4c4f-907b-dac7c0140f6c" (UID: "b7ee29a5-f392-4c4f-907b-dac7c0140f6c"). InnerVolumeSpecName "kube-api-access-p65vz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:02.825890 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.807630    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-hsg4m"]
Oct 30 19:36:02.828361 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.826392    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-8ffbj"]
Oct 30 19:36:02.830098 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.829972    5043 cpu_manager.go:414] "containerMap: removing container" podUID="9d198eae-9067-4789-b6b1-b8827ca742ab" containerName="payload-processing"
Oct 30 19:36:02.830260 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.830238    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d198eae-9067-4789-b6b1-b8827ca742ab" containerName="payload-processing"
Oct 30 19:36:02.830455 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.830440    5043 cpu_manager.go:414] "containerMap: removing container" podUID="b7ee29a5-f392-4c4f-907b-dac7c0140f6c" containerName="payload-pre-processing"
Oct 30 19:36:02.830548 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.830536    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7ee29a5-f392-4c4f-907b-dac7c0140f6c" containerName="payload-pre-processing"
Oct 30 19:36:02.830971 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.830913    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7ee29a5-f392-4c4f-907b-dac7c0140f6c" containerName="payload-pre-processing"
Oct 30 19:36:02.831119 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.831106    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d198eae-9067-4789-b6b1-b8827ca742ab" containerName="payload-processing"
Oct 30 19:36:02.858758 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.850867    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-8ffbj"
Oct 30 19:36:02.887880 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.844993    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-8ffbj"]
Oct 30 19:36:02.914026 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.907126    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/143f7a39-3eed-4dd0-9ef9-dc06ea5c4e4b-plugins-config-volume\") pod \"payload-processing-e2e-mt-20e66fa4-54557c4fd4-8ffbj\" (UID: \"143f7a39-3eed-4dd0-9ef9-dc06ea5c4e4b\") " pod="openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-8ffbj"
Oct 30 19:36:02.914405 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.914381    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhbzs\" (UniqueName: \"kubernetes.io/projected/143f7a39-3eed-4dd0-9ef9-dc06ea5c4e4b-kube-api-access-bhbzs\") pod \"payload-processing-e2e-mt-20e66fa4-54557c4fd4-8ffbj\" (UID: \"143f7a39-3eed-4dd0-9ef9-dc06ea5c4e4b\") " pod="openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-8ffbj"
Oct 30 19:36:02.914790 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.914771    5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7ee29a5-f392-4c4f-907b-dac7c0140f6c-plugins-config-volume\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:02.914909 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.914896    5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d198eae-9067-4789-b6b1-b8827ca742ab-plugins-config-volume\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:02.915000 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.914988    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p65vz\" (UniqueName: \"kubernetes.io/projected/b7ee29a5-f392-4c4f-907b-dac7c0140f6c-kube-api-access-p65vz\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:02.915095 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.915083    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r74lq\" (UniqueName: \"kubernetes.io/projected/9d198eae-9067-4789-b6b1-b8827ca742ab-kube-api-access-r74lq\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:02.936758 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.929573    5043 generic.go:356] "Generic (PLEG): container finished" podID="b7ee29a5-f392-4c4f-907b-dac7c0140f6c" containerID="1a739e75c29e65878454639cd97bc127d806eb8ccc45087f347ce0f17a32dd95" exitCode=0
Oct 30 19:36:02.936758 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.929882    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c7880b-c46dcb8b7-dkz2k" event={"ID":"b7ee29a5-f392-4c4f-907b-dac7c0140f6c","Type":"ContainerDied","Data":"1a739e75c29e65878454639cd97bc127d806eb8ccc45087f347ce0f17a32dd95"}
Oct 30 19:36:02.936758 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.929936    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c7880b-c46dcb8b7-dkz2k" event={"ID":"b7ee29a5-f392-4c4f-907b-dac7c0140f6c","Type":"ContainerDied","Data":"02587fe980b65c70fba0bbb589ecbb5d342edda7e0800bf69da9daaa47180b41"}
Oct 30 19:36:02.936758 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.929981    5043 scope.go:122] "RemoveContainer" containerID="1a739e75c29e65878454639cd97bc127d806eb8ccc45087f347ce0f17a32dd95"
Oct 30 19:36:02.936758 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.930157    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c7880b-c46dcb8b7-dkz2k"
Oct 30 19:36:02.949463 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.948900    5043 generic.go:356] "Generic (PLEG): container finished" podID="9d198eae-9067-4789-b6b1-b8827ca742ab" containerID="0ca3205e20388759dd42f2e1f7f6acc2fb9f01853c0924c43ed5f69560fd6937" exitCode=0
Oct 30 19:36:02.949463 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.948964    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c7880b-d5b8b5f94-gwr52"
Oct 30 19:36:02.949463 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.948999    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c7880b-d5b8b5f94-gwr52" event={"ID":"9d198eae-9067-4789-b6b1-b8827ca742ab","Type":"ContainerDied","Data":"0ca3205e20388759dd42f2e1f7f6acc2fb9f01853c0924c43ed5f69560fd6937"}
Oct 30 19:36:02.949463 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.949050    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c7880b-d5b8b5f94-gwr52" event={"ID":"9d198eae-9067-4789-b6b1-b8827ca742ab","Type":"ContainerDied","Data":"340842a1759bae884399dd726e072fe068b0d0a336dfbc54dab9bed50fc0fb87"}
Oct 30 19:36:02.978499 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:02.978471    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-hsg4m"
Oct 30 19:36:03.017510 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.017430    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/143f7a39-3eed-4dd0-9ef9-dc06ea5c4e4b-plugins-config-volume\") pod \"payload-processing-e2e-mt-20e66fa4-54557c4fd4-8ffbj\" (UID: \"143f7a39-3eed-4dd0-9ef9-dc06ea5c4e4b\") " pod="openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-8ffbj"
Oct 30 19:36:03.017686 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.017557    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhbzs\" (UniqueName: \"kubernetes.io/projected/143f7a39-3eed-4dd0-9ef9-dc06ea5c4e4b-kube-api-access-bhbzs\") pod \"payload-processing-e2e-mt-20e66fa4-54557c4fd4-8ffbj\" (UID: \"143f7a39-3eed-4dd0-9ef9-dc06ea5c4e4b\") " pod="openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-8ffbj"
Oct 30 19:36:03.018806 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.018585    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/143f7a39-3eed-4dd0-9ef9-dc06ea5c4e4b-plugins-config-volume\") pod \"payload-processing-e2e-mt-20e66fa4-54557c4fd4-8ffbj\" (UID: \"143f7a39-3eed-4dd0-9ef9-dc06ea5c4e4b\") " pod="openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-8ffbj"
Oct 30 19:36:03.031221 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.031189    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhbzs\" (UniqueName: \"kubernetes.io/projected/143f7a39-3eed-4dd0-9ef9-dc06ea5c4e4b-kube-api-access-bhbzs\") pod \"payload-processing-e2e-mt-20e66fa4-54557c4fd4-8ffbj\" (UID: \"143f7a39-3eed-4dd0-9ef9-dc06ea5c4e4b\") " pod="openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-8ffbj"
Oct 30 19:36:03.045878 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.034821    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10db525e-hsnq6"]
Oct 30 19:36:03.045878 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.043071    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10db525e-hsnq6"]
Oct 30 19:36:03.223717 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.218268    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-8ffbj"
Oct 30 19:36:03.230456 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.227236    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c7880b-d5b8b5f94-gwr52"]
Oct 30 19:36:03.238035 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.237998    5043 scope.go:122] "RemoveContainer" containerID="1a739e75c29e65878454639cd97bc127d806eb8ccc45087f347ce0f17a32dd95"
Oct 30 19:36:03.238903 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:36:03.238875    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a739e75c29e65878454639cd97bc127d806eb8ccc45087f347ce0f17a32dd95\": container with ID starting with 1a739e75c29e65878454639cd97bc127d806eb8ccc45087f347ce0f17a32dd95 not found: ID does not exist" containerID="1a739e75c29e65878454639cd97bc127d806eb8ccc45087f347ce0f17a32dd95"
Oct 30 19:36:03.239065 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.239034    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a739e75c29e65878454639cd97bc127d806eb8ccc45087f347ce0f17a32dd95"} err="failed to get container status \"1a739e75c29e65878454639cd97bc127d806eb8ccc45087f347ce0f17a32dd95\": rpc error: code = NotFound desc = could not find container \"1a739e75c29e65878454639cd97bc127d806eb8ccc45087f347ce0f17a32dd95\": container with ID starting with 1a739e75c29e65878454639cd97bc127d806eb8ccc45087f347ce0f17a32dd95 not found: ID does not exist"
Oct 30 19:36:03.239241 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.239226    5043 scope.go:122] "RemoveContainer" containerID="0ca3205e20388759dd42f2e1f7f6acc2fb9f01853c0924c43ed5f69560fd6937"
Oct 30 19:36:03.254938 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.254917    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c7880b-d5b8b5f94-gwr52"]
Oct 30 19:36:03.284206 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.284177    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c7880b-c46dcb8b7-dkz2k"]
Oct 30 19:36:03.292960 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.292234    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c7880b-c46dcb8b7-dkz2k"]
Oct 30 19:36:03.308939 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.308348    5043 scope.go:122] "RemoveContainer" containerID="0ca3205e20388759dd42f2e1f7f6acc2fb9f01853c0924c43ed5f69560fd6937"
Oct 30 19:36:03.308939 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:36:03.308787    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ca3205e20388759dd42f2e1f7f6acc2fb9f01853c0924c43ed5f69560fd6937\": container with ID starting with 0ca3205e20388759dd42f2e1f7f6acc2fb9f01853c0924c43ed5f69560fd6937 not found: ID does not exist" containerID="0ca3205e20388759dd42f2e1f7f6acc2fb9f01853c0924c43ed5f69560fd6937"
Oct 30 19:36:03.308939 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.308819    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ca3205e20388759dd42f2e1f7f6acc2fb9f01853c0924c43ed5f69560fd6937"} err="failed to get container status \"0ca3205e20388759dd42f2e1f7f6acc2fb9f01853c0924c43ed5f69560fd6937\": rpc error: code = NotFound desc = could not find container \"0ca3205e20388759dd42f2e1f7f6acc2fb9f01853c0924c43ed5f69560fd6937\": container with ID starting with 0ca3205e20388759dd42f2e1f7f6acc2fb9f01853c0924c43ed5f69560fd6937 not found: ID does not exist"
Oct 30 19:36:03.320312 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.320085    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq"
Oct 30 19:36:03.392511 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.392474    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv"
Oct 30 19:36:03.425622 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.421578    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/212ce985-524a-4391-8926-63aafdc58132-kube-api-access-tvsks\" (UniqueName: \"kubernetes.io/projected/212ce985-524a-4391-8926-63aafdc58132-kube-api-access-tvsks\") pod \"212ce985-524a-4391-8926-63aafdc58132\" (UID: \"212ce985-524a-4391-8926-63aafdc58132\") "
Oct 30 19:36:03.425622 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.421669    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23f8f6fc-85c3-46ac-88f5-08e08f246560-istio-token\" (UniqueName: \"kubernetes.io/projected/23f8f6fc-85c3-46ac-88f5-08e08f246560-istio-token\") pod \"23f8f6fc-85c3-46ac-88f5-08e08f246560\" (UID: \"23f8f6fc-85c3-46ac-88f5-08e08f246560\") "
Oct 30 19:36:03.425622 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.421719    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/212ce985-524a-4391-8926-63aafdc58132-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/212ce985-524a-4391-8926-63aafdc58132-istio-podinfo\") pod \"212ce985-524a-4391-8926-63aafdc58132\" (UID: \"212ce985-524a-4391-8926-63aafdc58132\") "
Oct 30 19:36:03.426996 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.426151    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/212ce985-524a-4391-8926-63aafdc58132-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/212ce985-524a-4391-8926-63aafdc58132-workload-certs\") pod \"212ce985-524a-4391-8926-63aafdc58132\" (UID: \"212ce985-524a-4391-8926-63aafdc58132\") "
Oct 30 19:36:03.426996 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.426331    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23f8f6fc-85c3-46ac-88f5-08e08f246560-kube-api-access-zj9xg\" (UniqueName: \"kubernetes.io/projected/23f8f6fc-85c3-46ac-88f5-08e08f246560-kube-api-access-zj9xg\") pod \"23f8f6fc-85c3-46ac-88f5-08e08f246560\" (UID: \"23f8f6fc-85c3-46ac-88f5-08e08f246560\") "
Oct 30 19:36:03.426996 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.426369    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/212ce985-524a-4391-8926-63aafdc58132-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/212ce985-524a-4391-8926-63aafdc58132-istio-data\") pod \"212ce985-524a-4391-8926-63aafdc58132\" (UID: \"212ce985-524a-4391-8926-63aafdc58132\") "
Oct 30 19:36:03.426996 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.426432    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/212ce985-524a-4391-8926-63aafdc58132-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/212ce985-524a-4391-8926-63aafdc58132-workload-socket\") pod \"212ce985-524a-4391-8926-63aafdc58132\" (UID: \"212ce985-524a-4391-8926-63aafdc58132\") "
Oct 30 19:36:03.426996 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.426468    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/212ce985-524a-4391-8926-63aafdc58132-istio-token\" (UniqueName: \"kubernetes.io/projected/212ce985-524a-4391-8926-63aafdc58132-istio-token\") pod \"212ce985-524a-4391-8926-63aafdc58132\" (UID: \"212ce985-524a-4391-8926-63aafdc58132\") "
Oct 30 19:36:03.426996 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.426515    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/212ce985-524a-4391-8926-63aafdc58132-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/212ce985-524a-4391-8926-63aafdc58132-credential-socket\") pod \"212ce985-524a-4391-8926-63aafdc58132\" (UID: \"212ce985-524a-4391-8926-63aafdc58132\") "
Oct 30 19:36:03.426996 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.426556    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23f8f6fc-85c3-46ac-88f5-08e08f246560-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f8f6fc-85c3-46ac-88f5-08e08f246560-workload-certs\") pod \"23f8f6fc-85c3-46ac-88f5-08e08f246560\" (UID: \"23f8f6fc-85c3-46ac-88f5-08e08f246560\") "
Oct 30 19:36:03.426996 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.426597    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23f8f6fc-85c3-46ac-88f5-08e08f246560-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f8f6fc-85c3-46ac-88f5-08e08f246560-istio-data\") pod \"23f8f6fc-85c3-46ac-88f5-08e08f246560\" (UID: \"23f8f6fc-85c3-46ac-88f5-08e08f246560\") "
Oct 30 19:36:03.426996 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.426689    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23f8f6fc-85c3-46ac-88f5-08e08f246560-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f8f6fc-85c3-46ac-88f5-08e08f246560-istio-podinfo\") pod \"23f8f6fc-85c3-46ac-88f5-08e08f246560\" (UID: \"23f8f6fc-85c3-46ac-88f5-08e08f246560\") "
Oct 30 19:36:03.426996 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.426722    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23f8f6fc-85c3-46ac-88f5-08e08f246560-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f8f6fc-85c3-46ac-88f5-08e08f246560-istiod-ca-cert\") pod \"23f8f6fc-85c3-46ac-88f5-08e08f246560\" (UID: \"23f8f6fc-85c3-46ac-88f5-08e08f246560\") "
Oct 30 19:36:03.426996 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.426808    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/212ce985-524a-4391-8926-63aafdc58132-kube-api-access-tvsks" pod "212ce985-524a-4391-8926-63aafdc58132" (UID: "212ce985-524a-4391-8926-63aafdc58132"). InnerVolumeSpecName "kube-api-access-tvsks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:03.426996 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.426817    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/212ce985-524a-4391-8926-63aafdc58132-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/212ce985-524a-4391-8926-63aafdc58132-istiod-ca-cert\") pod \"212ce985-524a-4391-8926-63aafdc58132\" (UID: \"212ce985-524a-4391-8926-63aafdc58132\") "
Oct 30 19:36:03.426996 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.426831    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/212ce985-524a-4391-8926-63aafdc58132-workload-certs" pod "212ce985-524a-4391-8926-63aafdc58132" (UID: "212ce985-524a-4391-8926-63aafdc58132"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:03.426996 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.426916    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23f8f6fc-85c3-46ac-88f5-08e08f246560-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f8f6fc-85c3-46ac-88f5-08e08f246560-istio-envoy\") pod \"23f8f6fc-85c3-46ac-88f5-08e08f246560\" (UID: \"23f8f6fc-85c3-46ac-88f5-08e08f246560\") "
Oct 30 19:36:03.426996 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.426999    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23f8f6fc-85c3-46ac-88f5-08e08f246560-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f8f6fc-85c3-46ac-88f5-08e08f246560-workload-socket\") pod \"23f8f6fc-85c3-46ac-88f5-08e08f246560\" (UID: \"23f8f6fc-85c3-46ac-88f5-08e08f246560\") "
Oct 30 19:36:03.427824 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.427032    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/212ce985-524a-4391-8926-63aafdc58132-credential-socket" pod "212ce985-524a-4391-8926-63aafdc58132" (UID: "212ce985-524a-4391-8926-63aafdc58132"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:03.427824 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.427065    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23f8f6fc-85c3-46ac-88f5-08e08f246560-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f8f6fc-85c3-46ac-88f5-08e08f246560-credential-socket\") pod \"23f8f6fc-85c3-46ac-88f5-08e08f246560\" (UID: \"23f8f6fc-85c3-46ac-88f5-08e08f246560\") "
Oct 30 19:36:03.427824 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.427160    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/212ce985-524a-4391-8926-63aafdc58132-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/212ce985-524a-4391-8926-63aafdc58132-istio-envoy\") pod \"212ce985-524a-4391-8926-63aafdc58132\" (UID: \"212ce985-524a-4391-8926-63aafdc58132\") "
Oct 30 19:36:03.427824 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.427501    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f8f6fc-85c3-46ac-88f5-08e08f246560-workload-socket" pod "23f8f6fc-85c3-46ac-88f5-08e08f246560" (UID: "23f8f6fc-85c3-46ac-88f5-08e08f246560"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:03.427824 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.427558    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f8f6fc-85c3-46ac-88f5-08e08f246560-istio-data" pod "23f8f6fc-85c3-46ac-88f5-08e08f246560" (UID: "23f8f6fc-85c3-46ac-88f5-08e08f246560"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:03.428302 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.428245    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/212ce985-524a-4391-8926-63aafdc58132-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:03.428302 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.428269    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f8f6fc-85c3-46ac-88f5-08e08f246560-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:03.428302 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.428283    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f8f6fc-85c3-46ac-88f5-08e08f246560-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:03.428302 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.428298    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvsks\" (UniqueName: \"kubernetes.io/projected/212ce985-524a-4391-8926-63aafdc58132-kube-api-access-tvsks\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:03.428647 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.428312    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/212ce985-524a-4391-8926-63aafdc58132-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:03.428647 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.428375    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f8f6fc-85c3-46ac-88f5-08e08f246560-credential-socket" pod "23f8f6fc-85c3-46ac-88f5-08e08f246560" (UID: "23f8f6fc-85c3-46ac-88f5-08e08f246560"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:03.428647 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.428491    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f8f6fc-85c3-46ac-88f5-08e08f246560-workload-certs" pod "23f8f6fc-85c3-46ac-88f5-08e08f246560" (UID: "23f8f6fc-85c3-46ac-88f5-08e08f246560"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:03.428960 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.428843    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23f8f6fc-85c3-46ac-88f5-08e08f246560-istiod-ca-cert" pod "23f8f6fc-85c3-46ac-88f5-08e08f246560" (UID: "23f8f6fc-85c3-46ac-88f5-08e08f246560"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:36:03.428960 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.428856    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/212ce985-524a-4391-8926-63aafdc58132-istiod-ca-cert" pod "212ce985-524a-4391-8926-63aafdc58132" (UID: "212ce985-524a-4391-8926-63aafdc58132"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:36:03.429173 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.429113    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/212ce985-524a-4391-8926-63aafdc58132-workload-socket" pod "212ce985-524a-4391-8926-63aafdc58132" (UID: "212ce985-524a-4391-8926-63aafdc58132"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:03.429317 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.429284    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/212ce985-524a-4391-8926-63aafdc58132-istio-data" pod "212ce985-524a-4391-8926-63aafdc58132" (UID: "212ce985-524a-4391-8926-63aafdc58132"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:03.429531 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.429499    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23f8f6fc-85c3-46ac-88f5-08e08f246560-kube-api-access-zj9xg" pod "23f8f6fc-85c3-46ac-88f5-08e08f246560" (UID: "23f8f6fc-85c3-46ac-88f5-08e08f246560"). InnerVolumeSpecName "kube-api-access-zj9xg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:03.430162 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.429789    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/212ce985-524a-4391-8926-63aafdc58132-istio-podinfo" pod "212ce985-524a-4391-8926-63aafdc58132" (UID: "212ce985-524a-4391-8926-63aafdc58132"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:36:03.431189 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.431167    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/212ce985-524a-4391-8926-63aafdc58132-istio-envoy" pod "212ce985-524a-4391-8926-63aafdc58132" (UID: "212ce985-524a-4391-8926-63aafdc58132"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:03.432436 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.432409    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23f8f6fc-85c3-46ac-88f5-08e08f246560-istio-token" pod "23f8f6fc-85c3-46ac-88f5-08e08f246560" (UID: "23f8f6fc-85c3-46ac-88f5-08e08f246560"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:03.432857 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.432832    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/212ce985-524a-4391-8926-63aafdc58132-istio-token" pod "212ce985-524a-4391-8926-63aafdc58132" (UID: "212ce985-524a-4391-8926-63aafdc58132"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:03.433079 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.433059    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f8f6fc-85c3-46ac-88f5-08e08f246560-istio-envoy" pod "23f8f6fc-85c3-46ac-88f5-08e08f246560" (UID: "23f8f6fc-85c3-46ac-88f5-08e08f246560"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:03.433234 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.433215    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23f8f6fc-85c3-46ac-88f5-08e08f246560-istio-podinfo" pod "23f8f6fc-85c3-46ac-88f5-08e08f246560" (UID: "23f8f6fc-85c3-46ac-88f5-08e08f246560"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:36:03.516181 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:36:03.516154    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe77a5ed_7a32_4350_b60c_d119d766eb1c.slice/crio-6cc0fd7c16312d5836da5a533193f6e63345031443b96eb7400843714e456c01 WatchSource:0}: Error finding container 6cc0fd7c16312d5836da5a533193f6e63345031443b96eb7400843714e456c01: Status 404 returned error can't find the container with id 6cc0fd7c16312d5836da5a533193f6e63345031443b96eb7400843714e456c01
Oct 30 19:36:03.516181 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.516168    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-hsg4m"]
Oct 30 19:36:03.531948 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.529873    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f8f6fc-85c3-46ac-88f5-08e08f246560-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:03.531948 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.529905    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f8f6fc-85c3-46ac-88f5-08e08f246560-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:03.531948 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.529919    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/212ce985-524a-4391-8926-63aafdc58132-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:03.531948 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.529933    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f8f6fc-85c3-46ac-88f5-08e08f246560-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:03.531948 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.529945    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f8f6fc-85c3-46ac-88f5-08e08f246560-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:03.531948 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.529957    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/212ce985-524a-4391-8926-63aafdc58132-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:03.531948 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.529968    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23f8f6fc-85c3-46ac-88f5-08e08f246560-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:03.531948 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.529980    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/212ce985-524a-4391-8926-63aafdc58132-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:03.531948 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.529992    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj9xg\" (UniqueName: \"kubernetes.io/projected/23f8f6fc-85c3-46ac-88f5-08e08f246560-kube-api-access-zj9xg\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:03.531948 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.530006    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/212ce985-524a-4391-8926-63aafdc58132-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:03.531948 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.530017    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/212ce985-524a-4391-8926-63aafdc58132-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:03.531948 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.530033    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/212ce985-524a-4391-8926-63aafdc58132-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:03.531948 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.530045    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f8f6fc-85c3-46ac-88f5-08e08f246560-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:03.597681 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:36:03.597650    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod143f7a39_3eed_4dd0_9ef9_dc06ea5c4e4b.slice/crio-86d2b585cc75223e2134d96e120a3971e9e85180eaed973ecc91a787c86eb35a WatchSource:0}: Error finding container 86d2b585cc75223e2134d96e120a3971e9e85180eaed973ecc91a787c86eb35a: Status 404 returned error can't find the container with id 86d2b585cc75223e2134d96e120a3971e9e85180eaed973ecc91a787c86eb35a
Oct 30 19:36:03.600474 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.600447    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-8ffbj"]
Oct 30 19:36:03.969656 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.969620    5043 generic.go:356] "Generic (PLEG): container finished" podID="212ce985-524a-4391-8926-63aafdc58132" containerID="35ffcde418b19f88d351e011ed9b566bfe3f6111991ee359720fd4947da07efb" exitCode=0
Oct 30 19:36:03.970013 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.969977    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq"
Oct 30 19:36:03.970013 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.969868    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq" event={"ID":"212ce985-524a-4391-8926-63aafdc58132","Type":"ContainerDied","Data":"35ffcde418b19f88d351e011ed9b566bfe3f6111991ee359720fd4947da07efb"}
Oct 30 19:36:03.970378 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.970129    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq" event={"ID":"212ce985-524a-4391-8926-63aafdc58132","Type":"ContainerDied","Data":"32348090f808fc56f6bedb793032dccfb75e599640dd82acdaaea6354824dd98"}
Oct 30 19:36:03.970378 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.970291    5043 scope.go:122] "RemoveContainer" containerID="35ffcde418b19f88d351e011ed9b566bfe3f6111991ee359720fd4947da07efb"
Oct 30 19:36:03.992218 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.992179    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-8ffbj" event={"ID":"143f7a39-3eed-4dd0-9ef9-dc06ea5c4e4b","Type":"ContainerStarted","Data":"e781240bbb84d0e8aeb3e2ccb468d45c71e6929e00cc851a8705f1c00139dc9e"}
Oct 30 19:36:03.992379 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.992247    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-8ffbj" event={"ID":"143f7a39-3eed-4dd0-9ef9-dc06ea5c4e4b","Type":"ContainerStarted","Data":"86d2b585cc75223e2134d96e120a3971e9e85180eaed973ecc91a787c86eb35a"}
Oct 30 19:36:03.992958 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:03.992936    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-8ffbj"
Oct 30 19:36:04.004429 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:04.004409    5043 scope.go:122] "RemoveContainer" containerID="35ffcde418b19f88d351e011ed9b566bfe3f6111991ee359720fd4947da07efb"
Oct 30 19:36:04.011210 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:36:04.011179    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35ffcde418b19f88d351e011ed9b566bfe3f6111991ee359720fd4947da07efb\": container with ID starting with 35ffcde418b19f88d351e011ed9b566bfe3f6111991ee359720fd4947da07efb not found: ID does not exist" containerID="35ffcde418b19f88d351e011ed9b566bfe3f6111991ee359720fd4947da07efb"
Oct 30 19:36:04.011416 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:04.011393    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35ffcde418b19f88d351e011ed9b566bfe3f6111991ee359720fd4947da07efb"} err="failed to get container status \"35ffcde418b19f88d351e011ed9b566bfe3f6111991ee359720fd4947da07efb\": rpc error: code = NotFound desc = could not find container \"35ffcde418b19f88d351e011ed9b566bfe3f6111991ee359720fd4947da07efb\": container with ID starting with 35ffcde418b19f88d351e011ed9b566bfe3f6111991ee359720fd4947da07efb not found: ID does not exist"
Oct 30 19:36:04.016253 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:04.016175    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-8ffbj" podStartSLOduration=2.015986348 podStartE2EDuration="2.015986348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:04.015158995 +0000 UTC m=+1183.526218961" watchObservedRunningTime="2026-10-30 19:36:04.015986348 +0000 UTC m=+1183.527046336"
Oct 30 19:36:04.026362 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:04.019904    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-hsg4m" event={"ID":"fe77a5ed-7a32-4350-b60c-d119d766eb1c","Type":"ContainerStarted","Data":"b0994e83f0d43cdec9fa02599b8b378d8c3f650007893ec44a2eaac852142c17"}
Oct 30 19:36:04.026362 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:04.019967    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-hsg4m" event={"ID":"fe77a5ed-7a32-4350-b60c-d119d766eb1c","Type":"ContainerStarted","Data":"6cc0fd7c16312d5836da5a533193f6e63345031443b96eb7400843714e456c01"}
Oct 30 19:36:04.026362 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:04.021228    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-hsg4m"
Oct 30 19:36:04.026362 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:04.024640    5043 generic.go:356] "Generic (PLEG): container finished" podID="23f8f6fc-85c3-46ac-88f5-08e08f246560" containerID="8b068dfe8707cc390ec583f8fa4bc639edaf1fdabf87227a1d758ab4b49d149e" exitCode=0
Oct 30 19:36:04.026362 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:04.024667    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv" event={"ID":"23f8f6fc-85c3-46ac-88f5-08e08f246560","Type":"ContainerDied","Data":"8b068dfe8707cc390ec583f8fa4bc639edaf1fdabf87227a1d758ab4b49d149e"}
Oct 30 19:36:04.026362 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:04.024695    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv" event={"ID":"23f8f6fc-85c3-46ac-88f5-08e08f246560","Type":"ContainerDied","Data":"c93938c7040ac388e19101d773e024ffb103524bff47e765f9503fe4047442d5"}
Oct 30 19:36:04.026362 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:04.024716    5043 scope.go:122] "RemoveContainer" containerID="8b068dfe8707cc390ec583f8fa4bc639edaf1fdabf87227a1d758ab4b49d149e"
Oct 30 19:36:04.026362 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:04.024828    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv"
Oct 30 19:36:04.046271 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:04.046237    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq"]
Oct 30 19:36:04.048672 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:04.048601    5043 scope.go:122] "RemoveContainer" containerID="8b068dfe8707cc390ec583f8fa4bc639edaf1fdabf87227a1d758ab4b49d149e"
Oct 30 19:36:04.049037 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:36:04.049015    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b068dfe8707cc390ec583f8fa4bc639edaf1fdabf87227a1d758ab4b49d149e\": container with ID starting with 8b068dfe8707cc390ec583f8fa4bc639edaf1fdabf87227a1d758ab4b49d149e not found: ID does not exist" containerID="8b068dfe8707cc390ec583f8fa4bc639edaf1fdabf87227a1d758ab4b49d149e"
Oct 30 19:36:04.049132 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:04.049050    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b068dfe8707cc390ec583f8fa4bc639edaf1fdabf87227a1d758ab4b49d149e"} err="failed to get container status \"8b068dfe8707cc390ec583f8fa4bc639edaf1fdabf87227a1d758ab4b49d149e\": rpc error: code = NotFound desc = could not find container \"8b068dfe8707cc390ec583f8fa4bc639edaf1fdabf87227a1d758ab4b49d149e\": container with ID starting with 8b068dfe8707cc390ec583f8fa4bc639edaf1fdabf87227a1d758ab4b49d149e not found: ID does not exist"
Oct 30 19:36:04.050317 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:04.050294    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-jgbnq"]
Oct 30 19:36:04.068910 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:04.068866    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-hsg4m" podStartSLOduration=2.06885256 podStartE2EDuration="2.06885256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:04.06402983 +0000 UTC m=+1183.575089794" watchObservedRunningTime="2026-10-30 19:36:04.06885256 +0000 UTC m=+1183.579912543"
Oct 30 19:36:04.087501 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:04.087475    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv"]
Oct 30 19:36:04.095058 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:04.095031    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-qxlwv"]
Oct 30 19:36:04.648935 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:04.648903    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="212ce985-524a-4391-8926-63aafdc58132" path="/var/lib/kubelet/pods/212ce985-524a-4391-8926-63aafdc58132/volumes"
Oct 30 19:36:04.649434 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:04.649419    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23f8f6fc-85c3-46ac-88f5-08e08f246560" path="/var/lib/kubelet/pods/23f8f6fc-85c3-46ac-88f5-08e08f246560/volumes"
Oct 30 19:36:04.649861 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:04.649847    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d198eae-9067-4789-b6b1-b8827ca742ab" path="/var/lib/kubelet/pods/9d198eae-9067-4789-b6b1-b8827ca742ab/volumes"
Oct 30 19:36:04.650544 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:04.650526    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7ee29a5-f392-4c4f-907b-dac7c0140f6c" path="/var/lib/kubelet/pods/b7ee29a5-f392-4c4f-907b-dac7c0140f6c/volumes"
Oct 30 19:36:04.650994 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:04.650980    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0590df0-bd23-418f-a4d1-e84f723de4bf" path="/var/lib/kubelet/pods/f0590df0-bd23-418f-a4d1-e84f723de4bf/volumes"
Oct 30 19:36:14.759952 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:14.759770    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2356d951-gm2s9"]
Oct 30 19:36:14.761870 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:14.761031    5043 cpu_manager.go:414] "containerMap: removing container" podUID="23f8f6fc-85c3-46ac-88f5-08e08f246560" containerName="istio-proxy"
Oct 30 19:36:14.761870 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:14.761056    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23f8f6fc-85c3-46ac-88f5-08e08f246560" containerName="istio-proxy"
Oct 30 19:36:14.761870 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:14.761104    5043 cpu_manager.go:414] "containerMap: removing container" podUID="212ce985-524a-4391-8926-63aafdc58132" containerName="istio-proxy"
Oct 30 19:36:14.761870 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:14.761112    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="212ce985-524a-4391-8926-63aafdc58132" containerName="istio-proxy"
Oct 30 19:36:14.761870 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:14.761347    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="23f8f6fc-85c3-46ac-88f5-08e08f246560" containerName="istio-proxy"
Oct 30 19:36:14.761870 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:14.761367    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="212ce985-524a-4391-8926-63aafdc58132" containerName="istio-proxy"
Oct 30 19:36:14.811050 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:14.811018    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2356d951-gm2s9"]
Oct 30 19:36:14.854177 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:14.812867    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2356d951-gm2s9"
Oct 30 19:36:14.884076 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:14.884012    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:36:14.884241 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:14.884087    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:36:14.884241 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:14.884112    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:36:14.884241 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:14.884173    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:36:14.974481 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:14.974435    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51fb5873-a661-491d-8437-aa1c915f126e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2356d951-gm2s9\" (UID: \"51fb5873-a661-491d-8437-aa1c915f126e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2356d951-gm2s9"
Oct 30 19:36:14.974707 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:14.974688    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51fb5873-a661-491d-8437-aa1c915f126e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2356d951-gm2s9\" (UID: \"51fb5873-a661-491d-8437-aa1c915f126e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2356d951-gm2s9"
Oct 30 19:36:14.974867 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:14.974789    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt76w\" (UniqueName: \"kubernetes.io/projected/51fb5873-a661-491d-8437-aa1c915f126e-kube-api-access-tt76w\") pod \"maas-api-revoke-sub-2356d951-gm2s9\" (UID: \"51fb5873-a661-491d-8437-aa1c915f126e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2356d951-gm2s9"
Oct 30 19:36:15.049428 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:15.049333    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-8ffbj"
Oct 30 19:36:15.079019 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:15.078992    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51fb5873-a661-491d-8437-aa1c915f126e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2356d951-gm2s9\" (UID: \"51fb5873-a661-491d-8437-aa1c915f126e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2356d951-gm2s9"
Oct 30 19:36:15.079476 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:15.079455    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51fb5873-a661-491d-8437-aa1c915f126e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2356d951-gm2s9\" (UID: \"51fb5873-a661-491d-8437-aa1c915f126e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2356d951-gm2s9"
Oct 30 19:36:15.080993 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:15.080972    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt76w\" (UniqueName: \"kubernetes.io/projected/51fb5873-a661-491d-8437-aa1c915f126e-kube-api-access-tt76w\") pod \"maas-api-revoke-sub-2356d951-gm2s9\" (UID: \"51fb5873-a661-491d-8437-aa1c915f126e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2356d951-gm2s9"
Oct 30 19:36:15.081580 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:15.080566    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51fb5873-a661-491d-8437-aa1c915f126e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2356d951-gm2s9\" (UID: \"51fb5873-a661-491d-8437-aa1c915f126e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2356d951-gm2s9"
Oct 30 19:36:15.090765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:15.090177    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51fb5873-a661-491d-8437-aa1c915f126e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2356d951-gm2s9\" (UID: \"51fb5873-a661-491d-8437-aa1c915f126e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2356d951-gm2s9"
Oct 30 19:36:15.119469 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:15.119430    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt76w\" (UniqueName: \"kubernetes.io/projected/51fb5873-a661-491d-8437-aa1c915f126e-kube-api-access-tt76w\") pod \"maas-api-revoke-sub-2356d951-gm2s9\" (UID: \"51fb5873-a661-491d-8437-aa1c915f126e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2356d951-gm2s9"
Oct 30 19:36:15.254023 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:15.253984    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2356d951-gm2s9"
Oct 30 19:36:15.589982 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:15.584403    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2356d951-gm2s9"]
Oct 30 19:36:16.075814 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:16.072508    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-hsg4m"
Oct 30 19:36:16.224914 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:16.224873    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2356d951-gm2s9" event={"ID":"51fb5873-a661-491d-8437-aa1c915f126e","Type":"ContainerStarted","Data":"f6ce6fed86dbcc7e7b213ffcef2f9892d2f80c36cbff65a1c123cb200129c37b"}
Oct 30 19:36:17.353816 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:17.351280    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2356d951-gm2s9" event={"ID":"51fb5873-a661-491d-8437-aa1c915f126e","Type":"ContainerStarted","Data":"64db22f423648649dd1d2d8b617043e30d9f6c4c17d9050cd98c4f6538405494"}
Oct 30 19:36:17.460805 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:17.456151    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2356d951-gm2s9" podStartSLOduration=3.456130375 podStartE2EDuration="3.456130375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:17.449717579 +0000 UTC m=+1196.960777566" watchObservedRunningTime="2026-10-30 19:36:17.456130375 +0000 UTC m=+1196.967190340"
Oct 30 19:36:18.372764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:18.372465    5043 generic.go:356] "Generic (PLEG): container finished" podID="51fb5873-a661-491d-8437-aa1c915f126e" containerID="64db22f423648649dd1d2d8b617043e30d9f6c4c17d9050cd98c4f6538405494" exitCode=0
Oct 30 19:36:18.372764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:18.372756    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2356d951-gm2s9" event={"ID":"51fb5873-a661-491d-8437-aa1c915f126e","Type":"ContainerDied","Data":"64db22f423648649dd1d2d8b617043e30d9f6c4c17d9050cd98c4f6538405494"}
Oct 30 19:36:19.673848 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:19.671951    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2356d951-gm2s9"
Oct 30 19:36:19.746894 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:19.745963    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51fb5873-a661-491d-8437-aa1c915f126e-kube-api-access-tt76w\" (UniqueName: \"kubernetes.io/projected/51fb5873-a661-491d-8437-aa1c915f126e-kube-api-access-tt76w\") pod \"51fb5873-a661-491d-8437-aa1c915f126e\" (UID: \"51fb5873-a661-491d-8437-aa1c915f126e\") "
Oct 30 19:36:19.746894 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:19.746213    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51fb5873-a661-491d-8437-aa1c915f126e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51fb5873-a661-491d-8437-aa1c915f126e-maas-api-openshift-ca\") pod \"51fb5873-a661-491d-8437-aa1c915f126e\" (UID: \"51fb5873-a661-491d-8437-aa1c915f126e\") "
Oct 30 19:36:19.746894 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:19.746333    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51fb5873-a661-491d-8437-aa1c915f126e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51fb5873-a661-491d-8437-aa1c915f126e-maas-api-xks-ca\") pod \"51fb5873-a661-491d-8437-aa1c915f126e\" (UID: \"51fb5873-a661-491d-8437-aa1c915f126e\") "
Oct 30 19:36:19.746894 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:19.746857    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51fb5873-a661-491d-8437-aa1c915f126e-maas-api-openshift-ca" pod "51fb5873-a661-491d-8437-aa1c915f126e" (UID: "51fb5873-a661-491d-8437-aa1c915f126e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:36:19.747274 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:19.747195    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51fb5873-a661-491d-8437-aa1c915f126e-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:19.749005 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:19.748973    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51fb5873-a661-491d-8437-aa1c915f126e-kube-api-access-tt76w" pod "51fb5873-a661-491d-8437-aa1c915f126e" (UID: "51fb5873-a661-491d-8437-aa1c915f126e"). InnerVolumeSpecName "kube-api-access-tt76w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:19.750156 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:19.750127    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51fb5873-a661-491d-8437-aa1c915f126e-maas-api-xks-ca" pod "51fb5873-a661-491d-8437-aa1c915f126e" (UID: "51fb5873-a661-491d-8437-aa1c915f126e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:36:19.859772 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:19.857608    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51fb5873-a661-491d-8437-aa1c915f126e-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:19.859772 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:19.857645    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt76w\" (UniqueName: \"kubernetes.io/projected/51fb5873-a661-491d-8437-aa1c915f126e-kube-api-access-tt76w\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:20.503830 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:20.500763    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2356d951-gm2s9" event={"ID":"51fb5873-a661-491d-8437-aa1c915f126e","Type":"ContainerDied","Data":"f6ce6fed86dbcc7e7b213ffcef2f9892d2f80c36cbff65a1c123cb200129c37b"}
Oct 30 19:36:20.503830 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:20.500834    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6ce6fed86dbcc7e7b213ffcef2f9892d2f80c36cbff65a1c123cb200129c37b"
Oct 30 19:36:20.503830 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:20.500916    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2356d951-gm2s9"
Oct 30 19:36:21.025264 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:21.025233    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cmw6w_76e21916-0744-4499-a0cb-64a7a44a8285/console-operator/2.log"
Oct 30 19:36:21.030989 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:21.030963    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cmw6w_76e21916-0744-4499-a0cb-64a7a44a8285/console-operator/2.log"
Oct 30 19:36:21.415558 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:21.415474    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5"]
Oct 30 19:36:21.416537 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:21.416509    5043 cpu_manager.go:414] "containerMap: removing container" podUID="51fb5873-a661-491d-8437-aa1c915f126e" containerName="revoke-keys"
Oct 30 19:36:21.416537 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:21.416537    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51fb5873-a661-491d-8437-aa1c915f126e" containerName="revoke-keys"
Oct 30 19:36:21.416825 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:21.416809    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="51fb5873-a661-491d-8437-aa1c915f126e" containerName="revoke-keys"
Oct 30 19:36:21.421196 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:21.421160    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5"
Oct 30 19:36:21.448324 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:21.448295    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-6a6348-openshift-default-dockercfg-bs2j2\""
Oct 30 19:36:21.590769 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:21.562158    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa769f06-2fc2-40ce-bb1a-a198b2274afb-istiod-ca-cert\") pod \"e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5\" (UID: \"aa769f06-2fc2-40ce-bb1a-a198b2274afb\") " pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5"
Oct 30 19:36:21.591169 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:21.591129    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa769f06-2fc2-40ce-bb1a-a198b2274afb-workload-certs\") pod \"e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5\" (UID: \"aa769f06-2fc2-40ce-bb1a-a198b2274afb\") " pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5"
Oct 30 19:36:21.591396 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:21.591374    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa769f06-2fc2-40ce-bb1a-a198b2274afb-credential-socket\") pod \"e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5\" (UID: \"aa769f06-2fc2-40ce-bb1a-a198b2274afb\") " pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5"
Oct 30 19:36:21.615763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:21.597866    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa769f06-2fc2-40ce-bb1a-a198b2274afb-istio-envoy\") pod \"e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5\" (UID: \"aa769f06-2fc2-40ce-bb1a-a198b2274afb\") " pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5"
Oct 30 19:36:21.615763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:21.597976    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa769f06-2fc2-40ce-bb1a-a198b2274afb-istio-token\") pod \"e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5\" (UID: \"aa769f06-2fc2-40ce-bb1a-a198b2274afb\") " pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5"
Oct 30 19:36:21.615763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:21.598094    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5sd8\" (UniqueName: \"kubernetes.io/projected/aa769f06-2fc2-40ce-bb1a-a198b2274afb-kube-api-access-q5sd8\") pod \"e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5\" (UID: \"aa769f06-2fc2-40ce-bb1a-a198b2274afb\") " pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5"
Oct 30 19:36:21.615763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:21.598164    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa769f06-2fc2-40ce-bb1a-a198b2274afb-istio-podinfo\") pod \"e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5\" (UID: \"aa769f06-2fc2-40ce-bb1a-a198b2274afb\") " pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5"
Oct 30 19:36:21.615763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:21.598318    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa769f06-2fc2-40ce-bb1a-a198b2274afb-workload-socket\") pod \"e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5\" (UID: \"aa769f06-2fc2-40ce-bb1a-a198b2274afb\") " pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5"
Oct 30 19:36:21.615763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:21.598508    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa769f06-2fc2-40ce-bb1a-a198b2274afb-istio-data\") pod \"e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5\" (UID: \"aa769f06-2fc2-40ce-bb1a-a198b2274afb\") " pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5"
Oct 30 19:36:21.816765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:21.751810    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa769f06-2fc2-40ce-bb1a-a198b2274afb-istio-data\") pod \"e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5\" (UID: \"aa769f06-2fc2-40ce-bb1a-a198b2274afb\") " pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5"
Oct 30 19:36:21.820763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:21.817066    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa769f06-2fc2-40ce-bb1a-a198b2274afb-istiod-ca-cert\") pod \"e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5\" (UID: \"aa769f06-2fc2-40ce-bb1a-a198b2274afb\") " pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5"
Oct 30 19:36:21.837763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:21.834594    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa769f06-2fc2-40ce-bb1a-a198b2274afb-workload-certs\") pod \"e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5\" (UID: \"aa769f06-2fc2-40ce-bb1a-a198b2274afb\") " pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5"
Oct 30 19:36:21.844062 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:21.844028    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa769f06-2fc2-40ce-bb1a-a198b2274afb-credential-socket\") pod \"e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5\" (UID: \"aa769f06-2fc2-40ce-bb1a-a198b2274afb\") " pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5"
Oct 30 19:36:21.861272 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:21.857912    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa769f06-2fc2-40ce-bb1a-a198b2274afb-istio-envoy\") pod \"e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5\" (UID: \"aa769f06-2fc2-40ce-bb1a-a198b2274afb\") " pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5"
Oct 30 19:36:21.861272 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:21.858039    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa769f06-2fc2-40ce-bb1a-a198b2274afb-istio-token\") pod \"e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5\" (UID: \"aa769f06-2fc2-40ce-bb1a-a198b2274afb\") " pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5"
Oct 30 19:36:21.861272 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:21.858166    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5sd8\" (UniqueName: \"kubernetes.io/projected/aa769f06-2fc2-40ce-bb1a-a198b2274afb-kube-api-access-q5sd8\") pod \"e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5\" (UID: \"aa769f06-2fc2-40ce-bb1a-a198b2274afb\") " pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5"
Oct 30 19:36:21.861272 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:21.858230    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa769f06-2fc2-40ce-bb1a-a198b2274afb-istio-podinfo\") pod \"e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5\" (UID: \"aa769f06-2fc2-40ce-bb1a-a198b2274afb\") " pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5"
Oct 30 19:36:21.861272 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:21.858383    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa769f06-2fc2-40ce-bb1a-a198b2274afb-workload-socket\") pod \"e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5\" (UID: \"aa769f06-2fc2-40ce-bb1a-a198b2274afb\") " pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5"
Oct 30 19:36:21.861272 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:21.858888    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa769f06-2fc2-40ce-bb1a-a198b2274afb-workload-socket\") pod \"e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5\" (UID: \"aa769f06-2fc2-40ce-bb1a-a198b2274afb\") " pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5"
Oct 30 19:36:21.861272 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:21.847167    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa769f06-2fc2-40ce-bb1a-a198b2274afb-workload-certs\") pod \"e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5\" (UID: \"aa769f06-2fc2-40ce-bb1a-a198b2274afb\") " pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5"
Oct 30 19:36:21.868469 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:21.861913    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa769f06-2fc2-40ce-bb1a-a198b2274afb-istiod-ca-cert\") pod \"e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5\" (UID: \"aa769f06-2fc2-40ce-bb1a-a198b2274afb\") " pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5"
Oct 30 19:36:21.868469 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:21.758401    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa769f06-2fc2-40ce-bb1a-a198b2274afb-istio-data\") pod \"e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5\" (UID: \"aa769f06-2fc2-40ce-bb1a-a198b2274afb\") " pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5"
Oct 30 19:36:21.889552 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:21.877016    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa769f06-2fc2-40ce-bb1a-a198b2274afb-credential-socket\") pod \"e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5\" (UID: \"aa769f06-2fc2-40ce-bb1a-a198b2274afb\") " pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5"
Oct 30 19:36:21.889552 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:21.877104    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5"]
Oct 30 19:36:21.889552 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:21.881755    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa769f06-2fc2-40ce-bb1a-a198b2274afb-istio-envoy\") pod \"e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5\" (UID: \"aa769f06-2fc2-40ce-bb1a-a198b2274afb\") " pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5"
Oct 30 19:36:21.999087 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:21.998710    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa769f06-2fc2-40ce-bb1a-a198b2274afb-istio-podinfo\") pod \"e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5\" (UID: \"aa769f06-2fc2-40ce-bb1a-a198b2274afb\") " pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5"
Oct 30 19:36:22.027189 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:22.005821    5043 scope.go:122] "RemoveContainer" containerID="85ed113461217a29b2913b30b659dd369fe6d289b2b04b938d3215a61cde4466"
Oct 30 19:36:22.027189 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:22.018707    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa769f06-2fc2-40ce-bb1a-a198b2274afb-istio-token\") pod \"e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5\" (UID: \"aa769f06-2fc2-40ce-bb1a-a198b2274afb\") " pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5"
Oct 30 19:36:22.078233 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:22.031811    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5sd8\" (UniqueName: \"kubernetes.io/projected/aa769f06-2fc2-40ce-bb1a-a198b2274afb-kube-api-access-q5sd8\") pod \"e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5\" (UID: \"aa769f06-2fc2-40ce-bb1a-a198b2274afb\") " pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5"
Oct 30 19:36:22.078233 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:22.076726    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5"
Oct 30 19:36:22.250520 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:22.250496    5043 scope.go:122] "RemoveContainer" containerID="cc85642a61c9eda036c2c852e0dd56906e127028bc6425b6aa4c5b10d0abf88c"
Oct 30 19:36:22.274023 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:22.267260    5043 scope.go:122] "RemoveContainer" containerID="412b9e685ac280b1a51d4da0545707f9c83b91ef14ae6d2d6a6d8f50e1f773ea"
Oct 30 19:36:22.328897 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:22.327316    5043 scope.go:122] "RemoveContainer" containerID="c0ad122ccd60265b8490ac4aef2015d653970d4ed114b26d3dd42fb77d0ab564"
Oct 30 19:36:22.348457 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:22.348026    5043 scope.go:122] "RemoveContainer" containerID="fa86e48bb627f66aa0fcde2d3507be7bbe3562f383c72591c651f7f540706172"
Oct 30 19:36:22.409518 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:22.406354    5043 scope.go:122] "RemoveContainer" containerID="36253c507721b59c5eeeef3086d50b1e3e2424f39eaef2e2eca94ba72696dc85"
Oct 30 19:36:22.434854 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:22.434196    5043 scope.go:122] "RemoveContainer" containerID="a9a5e6a0c82cd1f7e68ad57205fd17606e061133071f88bbdcbab4599150b367"
Oct 30 19:36:22.449513 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:22.449488    5043 scope.go:122] "RemoveContainer" containerID="73d3157577611f00bc912fdb50ebc5c22cf17d07e3f600e4c6983201fc5e9a08"
Oct 30 19:36:22.530082 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:22.521327    5043 scope.go:122] "RemoveContainer" containerID="50ca142dc7057ad4e9c2e502e02d8f7ab60638ef8ae5fefb3f13c9073fb60278"
Oct 30 19:36:22.543955 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:22.543928    5043 scope.go:122] "RemoveContainer" containerID="5d1c5fd409794239876528ad39b0b06ef0f78b809eb7c91d1d11132305727474"
Oct 30 19:36:22.619136 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:22.608964    5043 scope.go:122] "RemoveContainer" containerID="a09a74f4888b31d7f16381070f111e54a8a57610bcb588bb91c78bdca27dd91a"
Oct 30 19:36:22.631026 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:22.630985    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5"]
Oct 30 19:36:22.663121 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:22.663098    5043 scope.go:122] "RemoveContainer" containerID="f4d18cce38c8e51725329d076658f012568c5b4a2da12ea6823c383618c9aa64"
Oct 30 19:36:22.670788 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:22.665178    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:36:22.670788 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:22.665304    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:36:22.670788 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:22.665368    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:36:22.716027 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:22.716003    5043 scope.go:122] "RemoveContainer" containerID="f2ae8dfe297ea4a4abf4ad0228d55f2bdf54f44c5658e35f94afb802ab565450"
Oct 30 19:36:22.737955 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:22.737928    5043 scope.go:122] "RemoveContainer" containerID="665924c22ea9b263bf4f17c595d90ffae391803420e63f3e6064fdabed42bd47"
Oct 30 19:36:23.697136 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:23.697095    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5" event={"ID":"aa769f06-2fc2-40ce-bb1a-a198b2274afb","Type":"ContainerStarted","Data":"84556bc09334bd7c38da4f9ec3c9ba09f614085869b5048a421d05117f8210de"}
Oct 30 19:36:23.697669 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:23.697165    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5" event={"ID":"aa769f06-2fc2-40ce-bb1a-a198b2274afb","Type":"ContainerStarted","Data":"ac9ee860842ed64719bf87ec53dcee050d8a4c40e3668e3e607a9887ff0d4563"}
Oct 30 19:36:23.725781 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:23.725716    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5" podStartSLOduration=2.725702407 podStartE2EDuration="2.725702407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:23.720436007 +0000 UTC m=+1203.231495973" watchObservedRunningTime="2026-10-30 19:36:23.725702407 +0000 UTC m=+1203.236762371"
Oct 30 19:36:24.084095 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:24.083007    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5"
Oct 30 19:36:24.094769 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:24.093262    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5"
Oct 30 19:36:24.709975 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:24.709946    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5"
Oct 30 19:36:24.711060 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:24.711039    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5"
Oct 30 19:36:25.023453 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:36:25.023295    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:36:27.489761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:27.486789    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6a6348-79bcfb7655-cpxz9"]
Oct 30 19:36:27.555541 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:27.555509    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6a6348-79bcfb7655-cpxz9"]
Oct 30 19:36:27.555938 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:27.555920    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6a6348-677c897664-5q8vf"]
Oct 30 19:36:27.556162 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:27.555636    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a6348-79bcfb7655-cpxz9"
Oct 30 19:36:27.565091 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:27.565067    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-6a6348\""
Oct 30 19:36:27.566691 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:27.566671    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-6a6348-dockercfg-c2xr6\""
Oct 30 19:36:27.652590 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:27.609527    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s59n6\" (UniqueName: \"kubernetes.io/projected/0ff1108a-87ef-44f8-a85b-42e48bf99de9-kube-api-access-s59n6\") pod \"payload-pre-processing-e2e-api-b-6a6348-79bcfb7655-cpxz9\" (UID: \"0ff1108a-87ef-44f8-a85b-42e48bf99de9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a6348-79bcfb7655-cpxz9"
Oct 30 19:36:27.652590 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:27.609755    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ff1108a-87ef-44f8-a85b-42e48bf99de9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6a6348-79bcfb7655-cpxz9\" (UID: \"0ff1108a-87ef-44f8-a85b-42e48bf99de9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a6348-79bcfb7655-cpxz9"
Oct 30 19:36:27.712066 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:27.712017    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s59n6\" (UniqueName: \"kubernetes.io/projected/0ff1108a-87ef-44f8-a85b-42e48bf99de9-kube-api-access-s59n6\") pod \"payload-pre-processing-e2e-api-b-6a6348-79bcfb7655-cpxz9\" (UID: \"0ff1108a-87ef-44f8-a85b-42e48bf99de9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a6348-79bcfb7655-cpxz9"
Oct 30 19:36:27.712506 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:27.712226    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ff1108a-87ef-44f8-a85b-42e48bf99de9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6a6348-79bcfb7655-cpxz9\" (UID: \"0ff1108a-87ef-44f8-a85b-42e48bf99de9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a6348-79bcfb7655-cpxz9"
Oct 30 19:36:27.713563 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:27.713492    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ff1108a-87ef-44f8-a85b-42e48bf99de9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6a6348-79bcfb7655-cpxz9\" (UID: \"0ff1108a-87ef-44f8-a85b-42e48bf99de9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a6348-79bcfb7655-cpxz9"
Oct 30 19:36:27.791176 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:27.786459    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s59n6\" (UniqueName: \"kubernetes.io/projected/0ff1108a-87ef-44f8-a85b-42e48bf99de9-kube-api-access-s59n6\") pod \"payload-pre-processing-e2e-api-b-6a6348-79bcfb7655-cpxz9\" (UID: \"0ff1108a-87ef-44f8-a85b-42e48bf99de9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a6348-79bcfb7655-cpxz9"
Oct 30 19:36:27.851780 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:27.851105    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6a6348-677c897664-5q8vf"]
Oct 30 19:36:27.869951 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:27.863255    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6a6348-677c897664-5q8vf"
Oct 30 19:36:27.886231 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:27.886184    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a6348-79bcfb7655-cpxz9"
Oct 30 19:36:27.961987 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:27.947569    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59510403-878b-4f26-895c-cf3d22ef6ece-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6a6348-677c897664-5q8vf\" (UID: \"59510403-878b-4f26-895c-cf3d22ef6ece\") " pod="openshift-ingress/payload-processing-e2e-api-b-6a6348-677c897664-5q8vf"
Oct 30 19:36:27.961987 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:27.947636    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrfvv\" (UniqueName: \"kubernetes.io/projected/59510403-878b-4f26-895c-cf3d22ef6ece-kube-api-access-jrfvv\") pod \"payload-processing-e2e-api-b-6a6348-677c897664-5q8vf\" (UID: \"59510403-878b-4f26-895c-cf3d22ef6ece\") " pod="openshift-ingress/payload-processing-e2e-api-b-6a6348-677c897664-5q8vf"
Oct 30 19:36:28.049788 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:28.049381    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59510403-878b-4f26-895c-cf3d22ef6ece-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6a6348-677c897664-5q8vf\" (UID: \"59510403-878b-4f26-895c-cf3d22ef6ece\") " pod="openshift-ingress/payload-processing-e2e-api-b-6a6348-677c897664-5q8vf"
Oct 30 19:36:28.049788 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:28.049471    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrfvv\" (UniqueName: \"kubernetes.io/projected/59510403-878b-4f26-895c-cf3d22ef6ece-kube-api-access-jrfvv\") pod \"payload-processing-e2e-api-b-6a6348-677c897664-5q8vf\" (UID: \"59510403-878b-4f26-895c-cf3d22ef6ece\") " pod="openshift-ingress/payload-processing-e2e-api-b-6a6348-677c897664-5q8vf"
Oct 30 19:36:28.052015 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:28.051434    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59510403-878b-4f26-895c-cf3d22ef6ece-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6a6348-677c897664-5q8vf\" (UID: \"59510403-878b-4f26-895c-cf3d22ef6ece\") " pod="openshift-ingress/payload-processing-e2e-api-b-6a6348-677c897664-5q8vf"
Oct 30 19:36:28.180938 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:28.180903    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrfvv\" (UniqueName: \"kubernetes.io/projected/59510403-878b-4f26-895c-cf3d22ef6ece-kube-api-access-jrfvv\") pod \"payload-processing-e2e-api-b-6a6348-677c897664-5q8vf\" (UID: \"59510403-878b-4f26-895c-cf3d22ef6ece\") " pod="openshift-ingress/payload-processing-e2e-api-b-6a6348-677c897664-5q8vf"
Oct 30 19:36:28.322546 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:28.322199    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6a6348-677c897664-5q8vf"
Oct 30 19:36:28.327765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:28.327681    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c7880b-779lw"]
Oct 30 19:36:28.342407 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:28.342062    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c7880b-779lw"]
Oct 30 19:36:28.472670 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:36:28.472582    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ff1108a_87ef_44f8_a85b_42e48bf99de9.slice/crio-574cf2e65ffce7d2eed60fa9f63046c337236a287cb662df6bf2ffd57fdc7b11 WatchSource:0}: Error finding container 574cf2e65ffce7d2eed60fa9f63046c337236a287cb662df6bf2ffd57fdc7b11: Status 404 returned error can't find the container with id 574cf2e65ffce7d2eed60fa9f63046c337236a287cb662df6bf2ffd57fdc7b11
Oct 30 19:36:28.476589 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:28.476558    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6a6348-79bcfb7655-cpxz9"]
Oct 30 19:36:28.650596 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:28.650562    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61" path="/var/lib/kubelet/pods/129b9e3e-d6b6-40c3-adcc-ff16bcf3ca61/volumes"
Oct 30 19:36:28.664552 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:28.664523    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6a6348-677c897664-5q8vf"]
Oct 30 19:36:28.912274 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:28.912142    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6a6348-677c897664-5q8vf" event={"ID":"59510403-878b-4f26-895c-cf3d22ef6ece","Type":"ContainerStarted","Data":"11b1e0939f27f6704e7c2d53a6ffd0514e86d6e051d12707a9f417047c854afb"}
Oct 30 19:36:28.912274 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:28.912259    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6a6348-677c897664-5q8vf" event={"ID":"59510403-878b-4f26-895c-cf3d22ef6ece","Type":"ContainerStarted","Data":"2ff02125202640b796e77e870c07a75f289c314e03fa6a71d7fa7afc408b2ab6"}
Oct 30 19:36:28.913353 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:28.913325    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-6a6348-677c897664-5q8vf"
Oct 30 19:36:28.921976 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:28.921934    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a6348-79bcfb7655-cpxz9" event={"ID":"0ff1108a-87ef-44f8-a85b-42e48bf99de9","Type":"ContainerStarted","Data":"d6cbddd51e225a80e22333dcfd89dd23bdd21b5e654b43ca79065ec79da385e3"}
Oct 30 19:36:28.922141 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:28.921998    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a6348-79bcfb7655-cpxz9" event={"ID":"0ff1108a-87ef-44f8-a85b-42e48bf99de9","Type":"ContainerStarted","Data":"574cf2e65ffce7d2eed60fa9f63046c337236a287cb662df6bf2ffd57fdc7b11"}
Oct 30 19:36:28.922141 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:28.922127    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a6348-79bcfb7655-cpxz9"
Oct 30 19:36:28.930954 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:28.930900    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-6a6348-677c897664-5q8vf" podStartSLOduration=1.930883427 podStartE2EDuration="1.930883427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:28.929627766 +0000 UTC m=+1208.440687728" watchObservedRunningTime="2026-10-30 19:36:28.930883427 +0000 UTC m=+1208.441943393"
Oct 30 19:36:28.950665 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:28.950611    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a6348-79bcfb7655-cpxz9" podStartSLOduration=1.9505946920000001 podStartE2EDuration="1.950594692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:28.948155912 +0000 UTC m=+1208.459215878" watchObservedRunningTime="2026-10-30 19:36:28.950594692 +0000 UTC m=+1208.461654658"
Oct 30 19:36:29.583614 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:29.583527    5043 patch_prober.go:29] interesting pod/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 19:36:29.583614 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:29.583583    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk" podUID="c08cde99-b619-487f-b34f-3180ba5019be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 19:36:29.934952 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:29.934874    5043 generic.go:356] "Generic (PLEG): container finished" podID="c08cde99-b619-487f-b34f-3180ba5019be" containerID="a21e899d404b92394a64820357f8a6e4b28ee883343ff5bd273ae1759465914b" exitCode=0
Oct 30 19:36:29.935330 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:29.934947    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk" event={"ID":"c08cde99-b619-487f-b34f-3180ba5019be","Type":"ContainerDied","Data":"a21e899d404b92394a64820357f8a6e4b28ee883343ff5bd273ae1759465914b"}
Oct 30 19:36:29.935809 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:29.935781    5043 scope.go:122] "RemoveContainer" containerID="a21e899d404b92394a64820357f8a6e4b28ee883343ff5bd273ae1759465914b"
Oct 30 19:36:29.937372 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:29.937346    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:36:29.937454 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:29.937440    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:36:29.937503 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:29.937478    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:36:31.065764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:31.043688    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk" event={"ID":"c08cde99-b619-487f-b34f-3180ba5019be","Type":"ContainerStarted","Data":"e1fa23bc7e10f805bb6e9542b2743e6d7381d059afc293bd5967426fdbcc01d8"}
Oct 30 19:36:31.930178 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:31.930139    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk"
Oct 30 19:36:31.930567 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:31.930538    5043 patch_prober.go:29] interesting pod/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 19:36:31.930788 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:31.930759    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk" podUID="c08cde99-b619-487f-b34f-3180ba5019be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 19:36:32.099770 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:32.099717    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e66fa4-f72p2"]
Oct 30 19:36:32.105084 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:32.105061    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e66fa4-f72p2"
Oct 30 19:36:32.108999 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:32.108905    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:36:32.110496 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:32.109935    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:36:32.115170 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:32.114916    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:36:32.115170 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:32.115018    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:36:32.116716 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:32.116688    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e66fa4-f72p2"]
Oct 30 19:36:32.138258 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:32.137066    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60621b88-baeb-4d41-b665-7489d0c8ff7c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-20e66fa4-f72p2\" (UID: \"60621b88-baeb-4d41-b665-7489d0c8ff7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e66fa4-f72p2"
Oct 30 19:36:32.138258 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:32.137195    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhdvf\" (UniqueName: \"kubernetes.io/projected/60621b88-baeb-4d41-b665-7489d0c8ff7c-kube-api-access-mhdvf\") pod \"maas-api-revoke-keys-e2e-mt-20e66fa4-f72p2\" (UID: \"60621b88-baeb-4d41-b665-7489d0c8ff7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e66fa4-f72p2"
Oct 30 19:36:32.138258 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:32.137275    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60621b88-baeb-4d41-b665-7489d0c8ff7c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-20e66fa4-f72p2\" (UID: \"60621b88-baeb-4d41-b665-7489d0c8ff7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e66fa4-f72p2"
Oct 30 19:36:32.239540 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:32.239499    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60621b88-baeb-4d41-b665-7489d0c8ff7c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-20e66fa4-f72p2\" (UID: \"60621b88-baeb-4d41-b665-7489d0c8ff7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e66fa4-f72p2"
Oct 30 19:36:32.239707 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:32.239593    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhdvf\" (UniqueName: \"kubernetes.io/projected/60621b88-baeb-4d41-b665-7489d0c8ff7c-kube-api-access-mhdvf\") pod \"maas-api-revoke-keys-e2e-mt-20e66fa4-f72p2\" (UID: \"60621b88-baeb-4d41-b665-7489d0c8ff7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e66fa4-f72p2"
Oct 30 19:36:32.239707 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:32.239659    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60621b88-baeb-4d41-b665-7489d0c8ff7c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-20e66fa4-f72p2\" (UID: \"60621b88-baeb-4d41-b665-7489d0c8ff7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e66fa4-f72p2"
Oct 30 19:36:32.240320 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:32.240295    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60621b88-baeb-4d41-b665-7489d0c8ff7c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-20e66fa4-f72p2\" (UID: \"60621b88-baeb-4d41-b665-7489d0c8ff7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e66fa4-f72p2"
Oct 30 19:36:32.242144 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:32.242122    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60621b88-baeb-4d41-b665-7489d0c8ff7c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-20e66fa4-f72p2\" (UID: \"60621b88-baeb-4d41-b665-7489d0c8ff7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e66fa4-f72p2"
Oct 30 19:36:32.251368 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:32.251348    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhdvf\" (UniqueName: \"kubernetes.io/projected/60621b88-baeb-4d41-b665-7489d0c8ff7c-kube-api-access-mhdvf\") pod \"maas-api-revoke-keys-e2e-mt-20e66fa4-f72p2\" (UID: \"60621b88-baeb-4d41-b665-7489d0c8ff7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e66fa4-f72p2"
Oct 30 19:36:32.429841 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:32.429797    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e66fa4-f72p2"
Oct 30 19:36:32.801656 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:32.801621    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e66fa4-f72p2"]
Oct 30 19:36:32.802784 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:36:32.802713    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60621b88_baeb_4d41_b665_7489d0c8ff7c.slice/crio-c171e97e8b86e02af3df3eb180b7361626478aded93b065bc786dc00452f02a8 WatchSource:0}: Error finding container c171e97e8b86e02af3df3eb180b7361626478aded93b065bc786dc00452f02a8: Status 404 returned error can't find the container with id c171e97e8b86e02af3df3eb180b7361626478aded93b065bc786dc00452f02a8
Oct 30 19:36:32.918859 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:32.914206    5043 patch_prober.go:29] interesting pod/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 19:36:32.918859 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:32.914291    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk" podUID="c08cde99-b619-487f-b34f-3180ba5019be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 19:36:33.056334 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:33.056235    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e66fa4-f72p2" event={"ID":"60621b88-baeb-4d41-b665-7489d0c8ff7c","Type":"ContainerStarted","Data":"c171e97e8b86e02af3df3eb180b7361626478aded93b065bc786dc00452f02a8"}
Oct 30 19:36:33.914037 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:33.914003    5043 patch_prober.go:29] interesting pod/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 19:36:33.914449 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:33.914057    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk" podUID="c08cde99-b619-487f-b34f-3180ba5019be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 19:36:34.155498 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:34.155458    5043 generic.go:356] "Generic (PLEG): container finished" podID="60621b88-baeb-4d41-b665-7489d0c8ff7c" containerID="9a2fc964e42b10685d8154dc527ed1ca8d8411fc3aec890b938260d420107081" exitCode=0
Oct 30 19:36:34.168887 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:34.164101    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e66fa4-f72p2" event={"ID":"60621b88-baeb-4d41-b665-7489d0c8ff7c","Type":"ContainerDied","Data":"9a2fc964e42b10685d8154dc527ed1ca8d8411fc3aec890b938260d420107081"}
Oct 30 19:36:34.915825 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:34.915161    5043 patch_prober.go:29] interesting pod/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 19:36:34.915825 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:34.915221    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk" podUID="c08cde99-b619-487f-b34f-3180ba5019be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 19:36:35.504524 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:35.504443    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e66fa4-f72p2"
Oct 30 19:36:35.545250 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:35.545219    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60621b88-baeb-4d41-b665-7489d0c8ff7c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60621b88-baeb-4d41-b665-7489d0c8ff7c-maas-api-openshift-ca\") pod \"60621b88-baeb-4d41-b665-7489d0c8ff7c\" (UID: \"60621b88-baeb-4d41-b665-7489d0c8ff7c\") "
Oct 30 19:36:35.545656 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:35.545614    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60621b88-baeb-4d41-b665-7489d0c8ff7c-maas-api-openshift-ca" pod "60621b88-baeb-4d41-b665-7489d0c8ff7c" (UID: "60621b88-baeb-4d41-b665-7489d0c8ff7c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:36:35.545656 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:35.545627    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60621b88-baeb-4d41-b665-7489d0c8ff7c-kube-api-access-mhdvf\" (UniqueName: \"kubernetes.io/projected/60621b88-baeb-4d41-b665-7489d0c8ff7c-kube-api-access-mhdvf\") pod \"60621b88-baeb-4d41-b665-7489d0c8ff7c\" (UID: \"60621b88-baeb-4d41-b665-7489d0c8ff7c\") "
Oct 30 19:36:35.545859 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:35.545813    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60621b88-baeb-4d41-b665-7489d0c8ff7c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60621b88-baeb-4d41-b665-7489d0c8ff7c-maas-api-xks-ca\") pod \"60621b88-baeb-4d41-b665-7489d0c8ff7c\" (UID: \"60621b88-baeb-4d41-b665-7489d0c8ff7c\") "
Oct 30 19:36:35.546497 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:35.546475    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60621b88-baeb-4d41-b665-7489d0c8ff7c-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:35.548725 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:35.548689    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60621b88-baeb-4d41-b665-7489d0c8ff7c-maas-api-xks-ca" pod "60621b88-baeb-4d41-b665-7489d0c8ff7c" (UID: "60621b88-baeb-4d41-b665-7489d0c8ff7c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:36:35.549134 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:35.549108    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60621b88-baeb-4d41-b665-7489d0c8ff7c-kube-api-access-mhdvf" pod "60621b88-baeb-4d41-b665-7489d0c8ff7c" (UID: "60621b88-baeb-4d41-b665-7489d0c8ff7c"). InnerVolumeSpecName "kube-api-access-mhdvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:35.648697 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:35.648664    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhdvf\" (UniqueName: \"kubernetes.io/projected/60621b88-baeb-4d41-b665-7489d0c8ff7c-kube-api-access-mhdvf\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:35.648995 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:35.648965    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60621b88-baeb-4d41-b665-7489d0c8ff7c-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:35.949182 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:35.948171    5043 patch_prober.go:29] interesting pod/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 19:36:35.949182 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:35.948256    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk" podUID="c08cde99-b619-487f-b34f-3180ba5019be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 19:36:36.224780 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:36.224754    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e66fa4-f72p2"
Oct 30 19:36:36.224780 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:36.224766    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e66fa4-f72p2" event={"ID":"60621b88-baeb-4d41-b665-7489d0c8ff7c","Type":"ContainerDied","Data":"c171e97e8b86e02af3df3eb180b7361626478aded93b065bc786dc00452f02a8"}
Oct 30 19:36:36.225012 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:36.224813    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c171e97e8b86e02af3df3eb180b7361626478aded93b065bc786dc00452f02a8"
Oct 30 19:36:36.914190 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:36.914154    5043 patch_prober.go:29] interesting pod/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 19:36:36.914464 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:36.914436    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk" podUID="c08cde99-b619-487f-b34f-3180ba5019be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 19:36:37.343279 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:37.343249    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-95d2ac-ctr48"]
Oct 30 19:36:37.344123 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:37.344107    5043 cpu_manager.go:414] "containerMap: removing container" podUID="60621b88-baeb-4d41-b665-7489d0c8ff7c" containerName="revoke-keys"
Oct 30 19:36:37.344182 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:37.344125    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60621b88-baeb-4d41-b665-7489d0c8ff7c" containerName="revoke-keys"
Oct 30 19:36:37.344297 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:37.344287    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="60621b88-baeb-4d41-b665-7489d0c8ff7c" containerName="revoke-keys"
Oct 30 19:36:37.348200 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:37.348182    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-95d2ac-ctr48"
Oct 30 19:36:37.357165 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:37.357124    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-95d2ac-ctr48"]
Oct 30 19:36:37.358829 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:37.358806    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:36:37.359795 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:37.359716    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:36:37.359887 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:37.359807    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:36:37.359887 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:37.359833    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:36:37.423339 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:37.423304    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34027f03-ad24-4f28-b53d-8e21422dac2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-95d2ac-ctr48\" (UID: \"34027f03-ad24-4f28-b53d-8e21422dac2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-95d2ac-ctr48"
Oct 30 19:36:37.423549 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:37.423427    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34027f03-ad24-4f28-b53d-8e21422dac2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-95d2ac-ctr48\" (UID: \"34027f03-ad24-4f28-b53d-8e21422dac2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-95d2ac-ctr48"
Oct 30 19:36:37.423637 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:37.423616    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tks5\" (UniqueName: \"kubernetes.io/projected/34027f03-ad24-4f28-b53d-8e21422dac2e-kube-api-access-7tks5\") pod \"maas-api-revoke-keys-e2e-api-b-95d2ac-ctr48\" (UID: \"34027f03-ad24-4f28-b53d-8e21422dac2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-95d2ac-ctr48"
Oct 30 19:36:37.525308 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:37.525270    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tks5\" (UniqueName: \"kubernetes.io/projected/34027f03-ad24-4f28-b53d-8e21422dac2e-kube-api-access-7tks5\") pod \"maas-api-revoke-keys-e2e-api-b-95d2ac-ctr48\" (UID: \"34027f03-ad24-4f28-b53d-8e21422dac2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-95d2ac-ctr48"
Oct 30 19:36:37.525506 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:37.525424    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34027f03-ad24-4f28-b53d-8e21422dac2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-95d2ac-ctr48\" (UID: \"34027f03-ad24-4f28-b53d-8e21422dac2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-95d2ac-ctr48"
Oct 30 19:36:37.525589 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:37.525504    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34027f03-ad24-4f28-b53d-8e21422dac2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-95d2ac-ctr48\" (UID: \"34027f03-ad24-4f28-b53d-8e21422dac2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-95d2ac-ctr48"
Oct 30 19:36:37.526344 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:37.526255    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34027f03-ad24-4f28-b53d-8e21422dac2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-95d2ac-ctr48\" (UID: \"34027f03-ad24-4f28-b53d-8e21422dac2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-95d2ac-ctr48"
Oct 30 19:36:37.528196 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:37.528172    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34027f03-ad24-4f28-b53d-8e21422dac2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-95d2ac-ctr48\" (UID: \"34027f03-ad24-4f28-b53d-8e21422dac2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-95d2ac-ctr48"
Oct 30 19:36:37.536662 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:37.536640    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tks5\" (UniqueName: \"kubernetes.io/projected/34027f03-ad24-4f28-b53d-8e21422dac2e-kube-api-access-7tks5\") pod \"maas-api-revoke-keys-e2e-api-b-95d2ac-ctr48\" (UID: \"34027f03-ad24-4f28-b53d-8e21422dac2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-95d2ac-ctr48"
Oct 30 19:36:37.677561 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:37.677469    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-95d2ac-ctr48"
Oct 30 19:36:37.918161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:37.918129    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk"
Oct 30 19:36:37.918713 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:37.918688    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk"
Oct 30 19:36:37.919786 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:37.919760    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk"
Oct 30 19:36:38.015829 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:36:38.006532    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34027f03_ad24_4f28_b53d_8e21422dac2e.slice/crio-726d37f4f9f28779817d1f44189c3120c220870e4278225ebf42dc1b235e0b61 WatchSource:0}: Error finding container 726d37f4f9f28779817d1f44189c3120c220870e4278225ebf42dc1b235e0b61: Status 404 returned error can't find the container with id 726d37f4f9f28779817d1f44189c3120c220870e4278225ebf42dc1b235e0b61
Oct 30 19:36:38.015829 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:38.007985    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-95d2ac-ctr48"]
Oct 30 19:36:38.249791 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:38.249713    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-95d2ac-ctr48" event={"ID":"34027f03-ad24-4f28-b53d-8e21422dac2e","Type":"ContainerStarted","Data":"726d37f4f9f28779817d1f44189c3120c220870e4278225ebf42dc1b235e0b61"}
Oct 30 19:36:39.262580 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:39.262541    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-95d2ac-ctr48" event={"ID":"34027f03-ad24-4f28-b53d-8e21422dac2e","Type":"ContainerStarted","Data":"8e344b169ec915ed117875b39c9e69b1b7b0ee875e23b94b62b13eb037410182"}
Oct 30 19:36:39.284069 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:39.284019    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-95d2ac-ctr48" podStartSLOduration=2.284006697 podStartE2EDuration="2.284006697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:39.278613684 +0000 UTC m=+1218.789673651" watchObservedRunningTime="2026-10-30 19:36:39.284006697 +0000 UTC m=+1218.795066663"
Oct 30 19:36:39.938662 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:39.938633    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a6348-79bcfb7655-cpxz9"
Oct 30 19:36:40.275134 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:40.275089    5043 generic.go:356] "Generic (PLEG): container finished" podID="34027f03-ad24-4f28-b53d-8e21422dac2e" containerID="8e344b169ec915ed117875b39c9e69b1b7b0ee875e23b94b62b13eb037410182" exitCode=0
Oct 30 19:36:40.275591 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:40.275132    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-95d2ac-ctr48" event={"ID":"34027f03-ad24-4f28-b53d-8e21422dac2e","Type":"ContainerDied","Data":"8e344b169ec915ed117875b39c9e69b1b7b0ee875e23b94b62b13eb037410182"}
Oct 30 19:36:41.048822 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:41.048791    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-6a6348-677c897664-5q8vf"
Oct 30 19:36:41.513211 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:41.513182    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-95d2ac-ctr48"
Oct 30 19:36:41.547598 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:41.547565    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34027f03-ad24-4f28-b53d-8e21422dac2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34027f03-ad24-4f28-b53d-8e21422dac2e-maas-api-openshift-ca\") pod \"34027f03-ad24-4f28-b53d-8e21422dac2e\" (UID: \"34027f03-ad24-4f28-b53d-8e21422dac2e\") "
Oct 30 19:36:41.547925 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:41.547902    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34027f03-ad24-4f28-b53d-8e21422dac2e-kube-api-access-7tks5\" (UniqueName: \"kubernetes.io/projected/34027f03-ad24-4f28-b53d-8e21422dac2e-kube-api-access-7tks5\") pod \"34027f03-ad24-4f28-b53d-8e21422dac2e\" (UID: \"34027f03-ad24-4f28-b53d-8e21422dac2e\") "
Oct 30 19:36:41.548021 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:41.547996    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34027f03-ad24-4f28-b53d-8e21422dac2e-maas-api-openshift-ca" pod "34027f03-ad24-4f28-b53d-8e21422dac2e" (UID: "34027f03-ad24-4f28-b53d-8e21422dac2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:36:41.548100 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:41.548070    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34027f03-ad24-4f28-b53d-8e21422dac2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34027f03-ad24-4f28-b53d-8e21422dac2e-maas-api-xks-ca\") pod \"34027f03-ad24-4f28-b53d-8e21422dac2e\" (UID: \"34027f03-ad24-4f28-b53d-8e21422dac2e\") "
Oct 30 19:36:41.548947 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:41.548920    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34027f03-ad24-4f28-b53d-8e21422dac2e-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:41.550905 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:41.550874    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34027f03-ad24-4f28-b53d-8e21422dac2e-kube-api-access-7tks5" pod "34027f03-ad24-4f28-b53d-8e21422dac2e" (UID: "34027f03-ad24-4f28-b53d-8e21422dac2e"). InnerVolumeSpecName "kube-api-access-7tks5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:41.551028 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:41.550996    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34027f03-ad24-4f28-b53d-8e21422dac2e-maas-api-xks-ca" pod "34027f03-ad24-4f28-b53d-8e21422dac2e" (UID: "34027f03-ad24-4f28-b53d-8e21422dac2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:36:41.650447 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:41.650372    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34027f03-ad24-4f28-b53d-8e21422dac2e-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:41.650447 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:41.650401    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tks5\" (UniqueName: \"kubernetes.io/projected/34027f03-ad24-4f28-b53d-8e21422dac2e-kube-api-access-7tks5\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:42.300674 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:42.300646    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-95d2ac-ctr48"
Oct 30 19:36:42.300898 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:42.300686    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-95d2ac-ctr48" event={"ID":"34027f03-ad24-4f28-b53d-8e21422dac2e","Type":"ContainerDied","Data":"726d37f4f9f28779817d1f44189c3120c220870e4278225ebf42dc1b235e0b61"}
Oct 30 19:36:42.300898 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:42.300749    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="726d37f4f9f28779817d1f44189c3120c220870e4278225ebf42dc1b235e0b61"
Oct 30 19:36:42.311968 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:42.311941    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-8ffbj"]
Oct 30 19:36:42.332085 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:42.332054    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-hsg4m"]
Oct 30 19:36:43.380616 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:43.380584    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh"]
Oct 30 19:36:43.381169 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:43.381107    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh" podUID="0aa81e41-151e-47b9-8a09-f4cf20a52dad" containerName="istio-proxy" containerID="cri-o://4a572a7d72678e26f0fb808f9c6c84dbc1db2049e9f5185d0a0c6faf228a31aa" gracePeriod=30
Oct 30 19:36:43.392706 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:43.392670    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm"]
Oct 30 19:36:43.392992 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:43.392962    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm" podUID="03c6df78-8285-46c7-b5bd-896a0350c1de" containerName="istio-proxy" containerID="cri-o://92e838cda70530f6ebca3abe2d1d3da143d9c63d8038229c8fedefa8609345d7" gracePeriod=30
Oct 30 19:36:47.317762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:47.315811    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-8ffbj" podUID="143f7a39-3eed-4dd0-9ef9-dc06ea5c4e4b" containerName="payload-processing" containerID="cri-o://e781240bbb84d0e8aeb3e2ccb468d45c71e6929e00cc851a8705f1c00139dc9e" gracePeriod=25
Oct 30 19:36:47.332909 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:47.332838    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-hsg4m" podUID="fe77a5ed-7a32-4350-b60c-d119d766eb1c" containerName="payload-pre-processing" containerID="cri-o://b0994e83f0d43cdec9fa02599b8b378d8c3f650007893ec44a2eaac852142c17" gracePeriod=25
Oct 30 19:36:47.565975 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:47.565917    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-95d2ac-75f686bc4d-v7vw5"]
Oct 30 19:36:47.586126 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:47.585962    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-95d2ac-74dbfdbb99-fjbtf"]
Oct 30 19:36:47.690551 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:47.690520    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-8ffbj"
Oct 30 19:36:47.727284 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:47.727253    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-hsg4m"
Oct 30 19:36:47.740769 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:47.740702    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/143f7a39-3eed-4dd0-9ef9-dc06ea5c4e4b-kube-api-access-bhbzs\" (UniqueName: \"kubernetes.io/projected/143f7a39-3eed-4dd0-9ef9-dc06ea5c4e4b-kube-api-access-bhbzs\") pod \"143f7a39-3eed-4dd0-9ef9-dc06ea5c4e4b\" (UID: \"143f7a39-3eed-4dd0-9ef9-dc06ea5c4e4b\") "
Oct 30 19:36:47.742279 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:47.740979    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/143f7a39-3eed-4dd0-9ef9-dc06ea5c4e4b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/143f7a39-3eed-4dd0-9ef9-dc06ea5c4e4b-plugins-config-volume\") pod \"143f7a39-3eed-4dd0-9ef9-dc06ea5c4e4b\" (UID: \"143f7a39-3eed-4dd0-9ef9-dc06ea5c4e4b\") "
Oct 30 19:36:47.742279 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:47.741406    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/143f7a39-3eed-4dd0-9ef9-dc06ea5c4e4b-plugins-config-volume" pod "143f7a39-3eed-4dd0-9ef9-dc06ea5c4e4b" (UID: "143f7a39-3eed-4dd0-9ef9-dc06ea5c4e4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:36:47.742279 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:47.742161    5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/143f7a39-3eed-4dd0-9ef9-dc06ea5c4e4b-plugins-config-volume\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:47.744292 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:47.744269    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/143f7a39-3eed-4dd0-9ef9-dc06ea5c4e4b-kube-api-access-bhbzs" pod "143f7a39-3eed-4dd0-9ef9-dc06ea5c4e4b" (UID: "143f7a39-3eed-4dd0-9ef9-dc06ea5c4e4b"). InnerVolumeSpecName "kube-api-access-bhbzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:47.879807 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:47.878146    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhbzs\" (UniqueName: \"kubernetes.io/projected/143f7a39-3eed-4dd0-9ef9-dc06ea5c4e4b-kube-api-access-bhbzs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:48.113270 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.113238    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe77a5ed-7a32-4350-b60c-d119d766eb1c-kube-api-access-z6dbx\" (UniqueName: \"kubernetes.io/projected/fe77a5ed-7a32-4350-b60c-d119d766eb1c-kube-api-access-z6dbx\") pod \"fe77a5ed-7a32-4350-b60c-d119d766eb1c\" (UID: \"fe77a5ed-7a32-4350-b60c-d119d766eb1c\") "
Oct 30 19:36:48.114576 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.114379    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe77a5ed-7a32-4350-b60c-d119d766eb1c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe77a5ed-7a32-4350-b60c-d119d766eb1c-plugins-config-volume\") pod \"fe77a5ed-7a32-4350-b60c-d119d766eb1c\" (UID: \"fe77a5ed-7a32-4350-b60c-d119d766eb1c\") "
Oct 30 19:36:48.126656 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.124856    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe77a5ed-7a32-4350-b60c-d119d766eb1c-plugins-config-volume" pod "fe77a5ed-7a32-4350-b60c-d119d766eb1c" (UID: "fe77a5ed-7a32-4350-b60c-d119d766eb1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:36:48.146015 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.145922    5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe77a5ed-7a32-4350-b60c-d119d766eb1c-plugins-config-volume\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:48.161623 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.161497    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe77a5ed-7a32-4350-b60c-d119d766eb1c-kube-api-access-z6dbx" pod "fe77a5ed-7a32-4350-b60c-d119d766eb1c" (UID: "fe77a5ed-7a32-4350-b60c-d119d766eb1c"). InnerVolumeSpecName "kube-api-access-z6dbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:48.170564 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.170533    5043 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:36:48.170803 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.170780    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm" podUID="03c6df78-8285-46c7-b5bd-896a0350c1de" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:36:48.248726 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.248685    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6dbx\" (UniqueName: \"kubernetes.io/projected/fe77a5ed-7a32-4350-b60c-d119d766eb1c-kube-api-access-z6dbx\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:48.337324 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.337284    5043 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:36:48.337815 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.337357    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh" podUID="0aa81e41-151e-47b9-8a09-f4cf20a52dad" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:36:48.371116 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.371072    5043 generic.go:356] "Generic (PLEG): container finished" podID="143f7a39-3eed-4dd0-9ef9-dc06ea5c4e4b" containerID="e781240bbb84d0e8aeb3e2ccb468d45c71e6929e00cc851a8705f1c00139dc9e" exitCode=0
Oct 30 19:36:48.371356 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.371135    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-8ffbj"
Oct 30 19:36:48.371356 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.371240    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-8ffbj" event={"ID":"143f7a39-3eed-4dd0-9ef9-dc06ea5c4e4b","Type":"ContainerDied","Data":"e781240bbb84d0e8aeb3e2ccb468d45c71e6929e00cc851a8705f1c00139dc9e"}
Oct 30 19:36:48.371356 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.371324    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-8ffbj" event={"ID":"143f7a39-3eed-4dd0-9ef9-dc06ea5c4e4b","Type":"ContainerDied","Data":"86d2b585cc75223e2134d96e120a3971e9e85180eaed973ecc91a787c86eb35a"}
Oct 30 19:36:48.371540 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.371367    5043 scope.go:122] "RemoveContainer" containerID="e781240bbb84d0e8aeb3e2ccb468d45c71e6929e00cc851a8705f1c00139dc9e"
Oct 30 19:36:48.378046 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.378019    5043 generic.go:356] "Generic (PLEG): container finished" podID="fe77a5ed-7a32-4350-b60c-d119d766eb1c" containerID="b0994e83f0d43cdec9fa02599b8b378d8c3f650007893ec44a2eaac852142c17" exitCode=0
Oct 30 19:36:48.378201 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.378063    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-hsg4m"
Oct 30 19:36:48.378201 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.378112    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-hsg4m" event={"ID":"fe77a5ed-7a32-4350-b60c-d119d766eb1c","Type":"ContainerDied","Data":"b0994e83f0d43cdec9fa02599b8b378d8c3f650007893ec44a2eaac852142c17"}
Oct 30 19:36:48.378201 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.378166    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-hsg4m" event={"ID":"fe77a5ed-7a32-4350-b60c-d119d766eb1c","Type":"ContainerDied","Data":"6cc0fd7c16312d5836da5a533193f6e63345031443b96eb7400843714e456c01"}
Oct 30 19:36:48.390777 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.390715    5043 scope.go:122] "RemoveContainer" containerID="e781240bbb84d0e8aeb3e2ccb468d45c71e6929e00cc851a8705f1c00139dc9e"
Oct 30 19:36:48.394899 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:36:48.391890    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e781240bbb84d0e8aeb3e2ccb468d45c71e6929e00cc851a8705f1c00139dc9e\": container with ID starting with e781240bbb84d0e8aeb3e2ccb468d45c71e6929e00cc851a8705f1c00139dc9e not found: ID does not exist" containerID="e781240bbb84d0e8aeb3e2ccb468d45c71e6929e00cc851a8705f1c00139dc9e"
Oct 30 19:36:48.394899 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.391926    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e781240bbb84d0e8aeb3e2ccb468d45c71e6929e00cc851a8705f1c00139dc9e"} err="failed to get container status \"e781240bbb84d0e8aeb3e2ccb468d45c71e6929e00cc851a8705f1c00139dc9e\": rpc error: code = NotFound desc = could not find container \"e781240bbb84d0e8aeb3e2ccb468d45c71e6929e00cc851a8705f1c00139dc9e\": container with ID starting with e781240bbb84d0e8aeb3e2ccb468d45c71e6929e00cc851a8705f1c00139dc9e not found: ID does not exist"
Oct 30 19:36:48.394899 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.391949    5043 scope.go:122] "RemoveContainer" containerID="b0994e83f0d43cdec9fa02599b8b378d8c3f650007893ec44a2eaac852142c17"
Oct 30 19:36:48.407018 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.406794    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-8ffbj"]
Oct 30 19:36:48.433131 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.433061    5043 scope.go:122] "RemoveContainer" containerID="b0994e83f0d43cdec9fa02599b8b378d8c3f650007893ec44a2eaac852142c17"
Oct 30 19:36:48.434359 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:36:48.434276    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0994e83f0d43cdec9fa02599b8b378d8c3f650007893ec44a2eaac852142c17\": container with ID starting with b0994e83f0d43cdec9fa02599b8b378d8c3f650007893ec44a2eaac852142c17 not found: ID does not exist" containerID="b0994e83f0d43cdec9fa02599b8b378d8c3f650007893ec44a2eaac852142c17"
Oct 30 19:36:48.434359 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.434314    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0994e83f0d43cdec9fa02599b8b378d8c3f650007893ec44a2eaac852142c17"} err="failed to get container status \"b0994e83f0d43cdec9fa02599b8b378d8c3f650007893ec44a2eaac852142c17\": rpc error: code = NotFound desc = could not find container \"b0994e83f0d43cdec9fa02599b8b378d8c3f650007893ec44a2eaac852142c17\": container with ID starting with b0994e83f0d43cdec9fa02599b8b378d8c3f650007893ec44a2eaac852142c17 not found: ID does not exist"
Oct 30 19:36:48.441675 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.441646    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-20e66fa4-54557c4fd4-8ffbj"]
Oct 30 19:36:48.452259 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.452233    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-hsg4m"]
Oct 30 19:36:48.460975 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.460949    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-20e66fa4-997965c87-hsg4m"]
Oct 30 19:36:48.463917 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:36:48.463872    5043 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod143f7a39_3eed_4dd0_9ef9_dc06ea5c4e4b.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe77a5ed_7a32_4350_b60c_d119d766eb1c.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe77a5ed_7a32_4350_b60c_d119d766eb1c.slice/crio-6cc0fd7c16312d5836da5a533193f6e63345031443b96eb7400843714e456c01\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aa81e41_151e_47b9_8a09_f4cf20a52dad.slice/crio-4a572a7d72678e26f0fb808f9c6c84dbc1db2049e9f5185d0a0c6faf228a31aa.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod143f7a39_3eed_4dd0_9ef9_dc06ea5c4e4b.slice/crio-86d2b585cc75223e2134d96e120a3971e9e85180eaed973ecc91a787c86eb35a\": RecentStats: unable to find data in memory cache]"
Oct 30 19:36:48.676057 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.670957    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="143f7a39-3eed-4dd0-9ef9-dc06ea5c4e4b" path="/var/lib/kubelet/pods/143f7a39-3eed-4dd0-9ef9-dc06ea5c4e4b/volumes"
Oct 30 19:36:48.676057 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.671675    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe77a5ed-7a32-4350-b60c-d119d766eb1c" path="/var/lib/kubelet/pods/fe77a5ed-7a32-4350-b60c-d119d766eb1c/volumes"
Oct 30 19:36:48.794727 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.794696    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh"
Oct 30 19:36:48.872287 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.869703    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aa81e41-151e-47b9-8a09-f4cf20a52dad-kube-api-access-7mn8r\" (UniqueName: \"kubernetes.io/projected/0aa81e41-151e-47b9-8a09-f4cf20a52dad-kube-api-access-7mn8r\") pod \"0aa81e41-151e-47b9-8a09-f4cf20a52dad\" (UID: \"0aa81e41-151e-47b9-8a09-f4cf20a52dad\") "
Oct 30 19:36:48.872287 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.869850    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aa81e41-151e-47b9-8a09-f4cf20a52dad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aa81e41-151e-47b9-8a09-f4cf20a52dad-istio-envoy\") pod \"0aa81e41-151e-47b9-8a09-f4cf20a52dad\" (UID: \"0aa81e41-151e-47b9-8a09-f4cf20a52dad\") "
Oct 30 19:36:48.872287 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.869913    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aa81e41-151e-47b9-8a09-f4cf20a52dad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aa81e41-151e-47b9-8a09-f4cf20a52dad-istiod-ca-cert\") pod \"0aa81e41-151e-47b9-8a09-f4cf20a52dad\" (UID: \"0aa81e41-151e-47b9-8a09-f4cf20a52dad\") "
Oct 30 19:36:48.872287 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.869965    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aa81e41-151e-47b9-8a09-f4cf20a52dad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aa81e41-151e-47b9-8a09-f4cf20a52dad-workload-certs\") pod \"0aa81e41-151e-47b9-8a09-f4cf20a52dad\" (UID: \"0aa81e41-151e-47b9-8a09-f4cf20a52dad\") "
Oct 30 19:36:48.872287 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.870035    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aa81e41-151e-47b9-8a09-f4cf20a52dad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa81e41-151e-47b9-8a09-f4cf20a52dad-workload-socket\") pod \"0aa81e41-151e-47b9-8a09-f4cf20a52dad\" (UID: \"0aa81e41-151e-47b9-8a09-f4cf20a52dad\") "
Oct 30 19:36:48.872287 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.870250    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aa81e41-151e-47b9-8a09-f4cf20a52dad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aa81e41-151e-47b9-8a09-f4cf20a52dad-istio-data\") pod \"0aa81e41-151e-47b9-8a09-f4cf20a52dad\" (UID: \"0aa81e41-151e-47b9-8a09-f4cf20a52dad\") "
Oct 30 19:36:48.872287 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.870318    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0aa81e41-151e-47b9-8a09-f4cf20a52dad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aa81e41-151e-47b9-8a09-f4cf20a52dad-istio-podinfo\") pod \"0aa81e41-151e-47b9-8a09-f4cf20a52dad\" (UID: \"0aa81e41-151e-47b9-8a09-f4cf20a52dad\") "
Oct 30 19:36:48.872287 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.870445    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aa81e41-151e-47b9-8a09-f4cf20a52dad-istio-token\" (UniqueName: \"kubernetes.io/projected/0aa81e41-151e-47b9-8a09-f4cf20a52dad-istio-token\") pod \"0aa81e41-151e-47b9-8a09-f4cf20a52dad\" (UID: \"0aa81e41-151e-47b9-8a09-f4cf20a52dad\") "
Oct 30 19:36:48.872287 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.870520    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aa81e41-151e-47b9-8a09-f4cf20a52dad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa81e41-151e-47b9-8a09-f4cf20a52dad-credential-socket\") pod \"0aa81e41-151e-47b9-8a09-f4cf20a52dad\" (UID: \"0aa81e41-151e-47b9-8a09-f4cf20a52dad\") "
Oct 30 19:36:48.872287 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.871624    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aa81e41-151e-47b9-8a09-f4cf20a52dad-credential-socket" pod "0aa81e41-151e-47b9-8a09-f4cf20a52dad" (UID: "0aa81e41-151e-47b9-8a09-f4cf20a52dad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:48.872287 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.872163    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aa81e41-151e-47b9-8a09-f4cf20a52dad-workload-socket" pod "0aa81e41-151e-47b9-8a09-f4cf20a52dad" (UID: "0aa81e41-151e-47b9-8a09-f4cf20a52dad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:48.873289 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.873243    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aa81e41-151e-47b9-8a09-f4cf20a52dad-istio-data" pod "0aa81e41-151e-47b9-8a09-f4cf20a52dad" (UID: "0aa81e41-151e-47b9-8a09-f4cf20a52dad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:48.873289 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.873257    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aa81e41-151e-47b9-8a09-f4cf20a52dad-workload-certs" pod "0aa81e41-151e-47b9-8a09-f4cf20a52dad" (UID: "0aa81e41-151e-47b9-8a09-f4cf20a52dad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:48.873443 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.873334    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aa81e41-151e-47b9-8a09-f4cf20a52dad-istiod-ca-cert" pod "0aa81e41-151e-47b9-8a09-f4cf20a52dad" (UID: "0aa81e41-151e-47b9-8a09-f4cf20a52dad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:36:48.875006 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.874976    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aa81e41-151e-47b9-8a09-f4cf20a52dad-istio-envoy" pod "0aa81e41-151e-47b9-8a09-f4cf20a52dad" (UID: "0aa81e41-151e-47b9-8a09-f4cf20a52dad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:48.878336 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.876684    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aa81e41-151e-47b9-8a09-f4cf20a52dad-istio-token" pod "0aa81e41-151e-47b9-8a09-f4cf20a52dad" (UID: "0aa81e41-151e-47b9-8a09-f4cf20a52dad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:48.883235 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.883212    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm"
Oct 30 19:36:48.885548 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.885518    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0aa81e41-151e-47b9-8a09-f4cf20a52dad-istio-podinfo" pod "0aa81e41-151e-47b9-8a09-f4cf20a52dad" (UID: "0aa81e41-151e-47b9-8a09-f4cf20a52dad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:36:48.886143 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.886122    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aa81e41-151e-47b9-8a09-f4cf20a52dad-kube-api-access-7mn8r" pod "0aa81e41-151e-47b9-8a09-f4cf20a52dad" (UID: "0aa81e41-151e-47b9-8a09-f4cf20a52dad"). InnerVolumeSpecName "kube-api-access-7mn8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:48.972515 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.972482    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03c6df78-8285-46c7-b5bd-896a0350c1de-kube-api-access-qnvzf\" (UniqueName: \"kubernetes.io/projected/03c6df78-8285-46c7-b5bd-896a0350c1de-kube-api-access-qnvzf\") pod \"03c6df78-8285-46c7-b5bd-896a0350c1de\" (UID: \"03c6df78-8285-46c7-b5bd-896a0350c1de\") "
Oct 30 19:36:48.972688 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.972642    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03c6df78-8285-46c7-b5bd-896a0350c1de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03c6df78-8285-46c7-b5bd-896a0350c1de-istio-podinfo\") pod \"03c6df78-8285-46c7-b5bd-896a0350c1de\" (UID: \"03c6df78-8285-46c7-b5bd-896a0350c1de\") "
Oct 30 19:36:48.972809 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.972792    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03c6df78-8285-46c7-b5bd-896a0350c1de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03c6df78-8285-46c7-b5bd-896a0350c1de-workload-certs\") pod \"03c6df78-8285-46c7-b5bd-896a0350c1de\" (UID: \"03c6df78-8285-46c7-b5bd-896a0350c1de\") "
Oct 30 19:36:48.973128 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.973084    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03c6df78-8285-46c7-b5bd-896a0350c1de-workload-certs" pod "03c6df78-8285-46c7-b5bd-896a0350c1de" (UID: "03c6df78-8285-46c7-b5bd-896a0350c1de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:48.973424 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.973386    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03c6df78-8285-46c7-b5bd-896a0350c1de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c6df78-8285-46c7-b5bd-896a0350c1de-workload-socket\") pod \"03c6df78-8285-46c7-b5bd-896a0350c1de\" (UID: \"03c6df78-8285-46c7-b5bd-896a0350c1de\") "
Oct 30 19:36:48.973559 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.973453    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03c6df78-8285-46c7-b5bd-896a0350c1de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03c6df78-8285-46c7-b5bd-896a0350c1de-istio-envoy\") pod \"03c6df78-8285-46c7-b5bd-896a0350c1de\" (UID: \"03c6df78-8285-46c7-b5bd-896a0350c1de\") "
Oct 30 19:36:48.973559 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.973534    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03c6df78-8285-46c7-b5bd-896a0350c1de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03c6df78-8285-46c7-b5bd-896a0350c1de-istiod-ca-cert\") pod \"03c6df78-8285-46c7-b5bd-896a0350c1de\" (UID: \"03c6df78-8285-46c7-b5bd-896a0350c1de\") "
Oct 30 19:36:48.973683 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.973564    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03c6df78-8285-46c7-b5bd-896a0350c1de-istio-token\" (UniqueName: \"kubernetes.io/projected/03c6df78-8285-46c7-b5bd-896a0350c1de-istio-token\") pod \"03c6df78-8285-46c7-b5bd-896a0350c1de\" (UID: \"03c6df78-8285-46c7-b5bd-896a0350c1de\") "
Oct 30 19:36:48.973683 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.973659    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03c6df78-8285-46c7-b5bd-896a0350c1de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03c6df78-8285-46c7-b5bd-896a0350c1de-istio-data\") pod \"03c6df78-8285-46c7-b5bd-896a0350c1de\" (UID: \"03c6df78-8285-46c7-b5bd-896a0350c1de\") "
Oct 30 19:36:48.973829 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.973783    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03c6df78-8285-46c7-b5bd-896a0350c1de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c6df78-8285-46c7-b5bd-896a0350c1de-credential-socket\") pod \"03c6df78-8285-46c7-b5bd-896a0350c1de\" (UID: \"03c6df78-8285-46c7-b5bd-896a0350c1de\") "
Oct 30 19:36:48.974323 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.974277    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03c6df78-8285-46c7-b5bd-896a0350c1de-workload-socket" pod "03c6df78-8285-46c7-b5bd-896a0350c1de" (UID: "03c6df78-8285-46c7-b5bd-896a0350c1de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:48.974418 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.974333    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03c6df78-8285-46c7-b5bd-896a0350c1de-istiod-ca-cert" pod "03c6df78-8285-46c7-b5bd-896a0350c1de" (UID: "03c6df78-8285-46c7-b5bd-896a0350c1de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:36:48.975010 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.974962    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03c6df78-8285-46c7-b5bd-896a0350c1de-credential-socket" pod "03c6df78-8285-46c7-b5bd-896a0350c1de" (UID: "03c6df78-8285-46c7-b5bd-896a0350c1de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:48.975326 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.975293    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03c6df78-8285-46c7-b5bd-896a0350c1de-istio-data" pod "03c6df78-8285-46c7-b5bd-896a0350c1de" (UID: "03c6df78-8285-46c7-b5bd-896a0350c1de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:48.975996 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.975968    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03c6df78-8285-46c7-b5bd-896a0350c1de-kube-api-access-qnvzf" pod "03c6df78-8285-46c7-b5bd-896a0350c1de" (UID: "03c6df78-8285-46c7-b5bd-896a0350c1de"). InnerVolumeSpecName "kube-api-access-qnvzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:48.976379 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.976354    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03c6df78-8285-46c7-b5bd-896a0350c1de-istio-podinfo" pod "03c6df78-8285-46c7-b5bd-896a0350c1de" (UID: "03c6df78-8285-46c7-b5bd-896a0350c1de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:36:48.976716 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.976694    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aa81e41-151e-47b9-8a09-f4cf20a52dad-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:48.976900 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.976883    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aa81e41-151e-47b9-8a09-f4cf20a52dad-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:48.977008 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.976995    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03c6df78-8285-46c7-b5bd-896a0350c1de-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:48.977114 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.977091    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aa81e41-151e-47b9-8a09-f4cf20a52dad-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:48.977213 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.977119    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa81e41-151e-47b9-8a09-f4cf20a52dad-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:48.977213 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.977133    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c6df78-8285-46c7-b5bd-896a0350c1de-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:48.977213 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.977147    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03c6df78-8285-46c7-b5bd-896a0350c1de-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:48.977213 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.977162    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aa81e41-151e-47b9-8a09-f4cf20a52dad-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:48.977438 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.977293    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03c6df78-8285-46c7-b5bd-896a0350c1de-istio-token" pod "03c6df78-8285-46c7-b5bd-896a0350c1de" (UID: "03c6df78-8285-46c7-b5bd-896a0350c1de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:48.977438 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.977175    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03c6df78-8285-46c7-b5bd-896a0350c1de-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:48.977438 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.977340    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aa81e41-151e-47b9-8a09-f4cf20a52dad-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:48.977438 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.977353    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c6df78-8285-46c7-b5bd-896a0350c1de-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:48.977438 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.977367    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0aa81e41-151e-47b9-8a09-f4cf20a52dad-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:48.977438 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.977380    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa81e41-151e-47b9-8a09-f4cf20a52dad-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:48.977438 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.977392    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03c6df78-8285-46c7-b5bd-896a0350c1de-istio-envoy" pod "03c6df78-8285-46c7-b5bd-896a0350c1de" (UID: "03c6df78-8285-46c7-b5bd-896a0350c1de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:48.977438 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.977393    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnvzf\" (UniqueName: \"kubernetes.io/projected/03c6df78-8285-46c7-b5bd-896a0350c1de-kube-api-access-qnvzf\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:48.978451 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.977448    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mn8r\" (UniqueName: \"kubernetes.io/projected/0aa81e41-151e-47b9-8a09-f4cf20a52dad-kube-api-access-7mn8r\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:48.978451 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:48.977463    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03c6df78-8285-46c7-b5bd-896a0350c1de-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:49.079840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:49.079806    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03c6df78-8285-46c7-b5bd-896a0350c1de-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:49.079840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:49.079839    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03c6df78-8285-46c7-b5bd-896a0350c1de-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:49.391692 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:49.391651    5043 generic.go:356] "Generic (PLEG): container finished" podID="0aa81e41-151e-47b9-8a09-f4cf20a52dad" containerID="4a572a7d72678e26f0fb808f9c6c84dbc1db2049e9f5185d0a0c6faf228a31aa" exitCode=0
Oct 30 19:36:49.392188 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:49.391700    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh"
Oct 30 19:36:49.392188 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:49.391729    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh" event={"ID":"0aa81e41-151e-47b9-8a09-f4cf20a52dad","Type":"ContainerDied","Data":"4a572a7d72678e26f0fb808f9c6c84dbc1db2049e9f5185d0a0c6faf228a31aa"}
Oct 30 19:36:49.392188 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:49.391838    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh" event={"ID":"0aa81e41-151e-47b9-8a09-f4cf20a52dad","Type":"ContainerDied","Data":"88eed1fe5c7124798357ddbb5a205136528dccd701fc104ea2c5fea51d3c138b"}
Oct 30 19:36:49.392188 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:49.391861    5043 scope.go:122] "RemoveContainer" containerID="4a572a7d72678e26f0fb808f9c6c84dbc1db2049e9f5185d0a0c6faf228a31aa"
Oct 30 19:36:49.395041 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:49.395003    5043 generic.go:356] "Generic (PLEG): container finished" podID="03c6df78-8285-46c7-b5bd-896a0350c1de" containerID="92e838cda70530f6ebca3abe2d1d3da143d9c63d8038229c8fedefa8609345d7" exitCode=0
Oct 30 19:36:49.395183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:49.395077    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm"
Oct 30 19:36:49.395183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:49.395141    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm" event={"ID":"03c6df78-8285-46c7-b5bd-896a0350c1de","Type":"ContainerDied","Data":"92e838cda70530f6ebca3abe2d1d3da143d9c63d8038229c8fedefa8609345d7"}
Oct 30 19:36:49.395300 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:49.395192    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm" event={"ID":"03c6df78-8285-46c7-b5bd-896a0350c1de","Type":"ContainerDied","Data":"4fe589a72875ebe06416f6c51d18b8698e3a515be49eca928beaac94b5ab821a"}
Oct 30 19:36:49.425892 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:49.404890    5043 scope.go:122] "RemoveContainer" containerID="4a572a7d72678e26f0fb808f9c6c84dbc1db2049e9f5185d0a0c6faf228a31aa"
Oct 30 19:36:49.425892 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:36:49.405209    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a572a7d72678e26f0fb808f9c6c84dbc1db2049e9f5185d0a0c6faf228a31aa\": container with ID starting with 4a572a7d72678e26f0fb808f9c6c84dbc1db2049e9f5185d0a0c6faf228a31aa not found: ID does not exist" containerID="4a572a7d72678e26f0fb808f9c6c84dbc1db2049e9f5185d0a0c6faf228a31aa"
Oct 30 19:36:49.425892 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:49.405231    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a572a7d72678e26f0fb808f9c6c84dbc1db2049e9f5185d0a0c6faf228a31aa"} err="failed to get container status \"4a572a7d72678e26f0fb808f9c6c84dbc1db2049e9f5185d0a0c6faf228a31aa\": rpc error: code = NotFound desc = could not find container \"4a572a7d72678e26f0fb808f9c6c84dbc1db2049e9f5185d0a0c6faf228a31aa\": container with ID starting with 4a572a7d72678e26f0fb808f9c6c84dbc1db2049e9f5185d0a0c6faf228a31aa not found: ID does not exist"
Oct 30 19:36:49.425892 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:49.405257    5043 scope.go:122] "RemoveContainer" containerID="92e838cda70530f6ebca3abe2d1d3da143d9c63d8038229c8fedefa8609345d7"
Oct 30 19:36:49.476056 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:49.461580    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh"]
Oct 30 19:36:49.476698 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:49.476671    5043 scope.go:122] "RemoveContainer" containerID="92e838cda70530f6ebca3abe2d1d3da143d9c63d8038229c8fedefa8609345d7"
Oct 30 19:36:49.477059 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:36:49.477031    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92e838cda70530f6ebca3abe2d1d3da143d9c63d8038229c8fedefa8609345d7\": container with ID starting with 92e838cda70530f6ebca3abe2d1d3da143d9c63d8038229c8fedefa8609345d7 not found: ID does not exist" containerID="92e838cda70530f6ebca3abe2d1d3da143d9c63d8038229c8fedefa8609345d7"
Oct 30 19:36:49.477202 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:49.477071    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92e838cda70530f6ebca3abe2d1d3da143d9c63d8038229c8fedefa8609345d7"} err="failed to get container status \"92e838cda70530f6ebca3abe2d1d3da143d9c63d8038229c8fedefa8609345d7\": rpc error: code = NotFound desc = could not find container \"92e838cda70530f6ebca3abe2d1d3da143d9c63d8038229c8fedefa8609345d7\": container with ID starting with 92e838cda70530f6ebca3abe2d1d3da143d9c63d8038229c8fedefa8609345d7 not found: ID does not exist"
Oct 30 19:36:49.483442 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:49.482878    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t8psh"]
Oct 30 19:36:49.499570 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:49.496969    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm"]
Oct 30 19:36:49.507200 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:49.507169    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-qj2fm"]
Oct 30 19:36:50.648375 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:50.648334    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03c6df78-8285-46c7-b5bd-896a0350c1de" path="/var/lib/kubelet/pods/03c6df78-8285-46c7-b5bd-896a0350c1de/volumes"
Oct 30 19:36:50.649060 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:50.649038    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0aa81e41-151e-47b9-8a09-f4cf20a52dad" path="/var/lib/kubelet/pods/0aa81e41-151e-47b9-8a09-f4cf20a52dad/volumes"
Oct 30 19:36:52.568493 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:52.568424    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-95d2ac-75f686bc4d-v7vw5" podUID="6e5c13a6-0354-4d98-991a-ceb6ea271fb1" containerName="payload-processing" containerID="cri-o://fbd6b833037b34de08ecd362089deac03d9e2ae8b28ee1b5484d849860bb8b7a" gracePeriod=25
Oct 30 19:36:52.586997 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:52.586953    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-95d2ac-74dbfdbb99-fjbtf" podUID="2133d6b7-b08a-46ac-b888-2248e3d9e981" containerName="payload-pre-processing" containerID="cri-o://96952b99923f24adae5e9bce715087dafd62bee07d71fa923cba2aafdc421eb1" gracePeriod=25
Oct 30 19:36:52.954599 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:52.954205    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-95d2ac-75f686bc4d-v7vw5"
Oct 30 19:36:52.995390 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:52.995361    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-95d2ac-74dbfdbb99-fjbtf"
Oct 30 19:36:53.090589 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:53.090558    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2133d6b7-b08a-46ac-b888-2248e3d9e981-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2133d6b7-b08a-46ac-b888-2248e3d9e981-plugins-config-volume\") pod \"2133d6b7-b08a-46ac-b888-2248e3d9e981\" (UID: \"2133d6b7-b08a-46ac-b888-2248e3d9e981\") "
Oct 30 19:36:53.090809 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:53.090653    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e5c13a6-0354-4d98-991a-ceb6ea271fb1-kube-api-access-jfhr6\" (UniqueName: \"kubernetes.io/projected/6e5c13a6-0354-4d98-991a-ceb6ea271fb1-kube-api-access-jfhr6\") pod \"6e5c13a6-0354-4d98-991a-ceb6ea271fb1\" (UID: \"6e5c13a6-0354-4d98-991a-ceb6ea271fb1\") "
Oct 30 19:36:53.090809 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:53.090722    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2133d6b7-b08a-46ac-b888-2248e3d9e981-kube-api-access-pgn92\" (UniqueName: \"kubernetes.io/projected/2133d6b7-b08a-46ac-b888-2248e3d9e981-kube-api-access-pgn92\") pod \"2133d6b7-b08a-46ac-b888-2248e3d9e981\" (UID: \"2133d6b7-b08a-46ac-b888-2248e3d9e981\") "
Oct 30 19:36:53.090938 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:53.090871    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e5c13a6-0354-4d98-991a-ceb6ea271fb1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e5c13a6-0354-4d98-991a-ceb6ea271fb1-plugins-config-volume\") pod \"6e5c13a6-0354-4d98-991a-ceb6ea271fb1\" (UID: \"6e5c13a6-0354-4d98-991a-ceb6ea271fb1\") "
Oct 30 19:36:53.091067 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:53.091039    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2133d6b7-b08a-46ac-b888-2248e3d9e981-plugins-config-volume" pod "2133d6b7-b08a-46ac-b888-2248e3d9e981" (UID: "2133d6b7-b08a-46ac-b888-2248e3d9e981"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:36:53.091319 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:53.091293    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e5c13a6-0354-4d98-991a-ceb6ea271fb1-plugins-config-volume" pod "6e5c13a6-0354-4d98-991a-ceb6ea271fb1" (UID: "6e5c13a6-0354-4d98-991a-ceb6ea271fb1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:36:53.092280 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:53.092249    5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2133d6b7-b08a-46ac-b888-2248e3d9e981-plugins-config-volume\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:53.092407 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:53.092286    5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e5c13a6-0354-4d98-991a-ceb6ea271fb1-plugins-config-volume\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:53.093525 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:53.093504    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2133d6b7-b08a-46ac-b888-2248e3d9e981-kube-api-access-pgn92" pod "2133d6b7-b08a-46ac-b888-2248e3d9e981" (UID: "2133d6b7-b08a-46ac-b888-2248e3d9e981"). InnerVolumeSpecName "kube-api-access-pgn92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:53.093617 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:53.093544    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e5c13a6-0354-4d98-991a-ceb6ea271fb1-kube-api-access-jfhr6" pod "6e5c13a6-0354-4d98-991a-ceb6ea271fb1" (UID: "6e5c13a6-0354-4d98-991a-ceb6ea271fb1"). InnerVolumeSpecName "kube-api-access-jfhr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:53.194807 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:53.194685    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfhr6\" (UniqueName: \"kubernetes.io/projected/6e5c13a6-0354-4d98-991a-ceb6ea271fb1-kube-api-access-jfhr6\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:53.194807 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:53.194721    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgn92\" (UniqueName: \"kubernetes.io/projected/2133d6b7-b08a-46ac-b888-2248e3d9e981-kube-api-access-pgn92\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:53.450060 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:53.449973    5043 generic.go:356] "Generic (PLEG): container finished" podID="6e5c13a6-0354-4d98-991a-ceb6ea271fb1" containerID="fbd6b833037b34de08ecd362089deac03d9e2ae8b28ee1b5484d849860bb8b7a" exitCode=0
Oct 30 19:36:53.450345 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:53.450322    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-95d2ac-75f686bc4d-v7vw5" event={"ID":"6e5c13a6-0354-4d98-991a-ceb6ea271fb1","Type":"ContainerDied","Data":"fbd6b833037b34de08ecd362089deac03d9e2ae8b28ee1b5484d849860bb8b7a"}
Oct 30 19:36:53.450448 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:53.450384    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-95d2ac-75f686bc4d-v7vw5" event={"ID":"6e5c13a6-0354-4d98-991a-ceb6ea271fb1","Type":"ContainerDied","Data":"50ba9a4e3f9991d3004df87c466e99e2b7e2df2c9d90e4b328259ab93d7c31ea"}
Oct 30 19:36:53.450448 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:53.450416    5043 scope.go:122] "RemoveContainer" containerID="fbd6b833037b34de08ecd362089deac03d9e2ae8b28ee1b5484d849860bb8b7a"
Oct 30 19:36:53.450580 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:53.450566    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-95d2ac-75f686bc4d-v7vw5"
Oct 30 19:36:53.453891 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:53.453861    5043 generic.go:356] "Generic (PLEG): container finished" podID="2133d6b7-b08a-46ac-b888-2248e3d9e981" containerID="96952b99923f24adae5e9bce715087dafd62bee07d71fa923cba2aafdc421eb1" exitCode=0
Oct 30 19:36:53.454053 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:53.453907    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-95d2ac-74dbfdbb99-fjbtf"
Oct 30 19:36:53.454053 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:53.453960    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-95d2ac-74dbfdbb99-fjbtf" event={"ID":"2133d6b7-b08a-46ac-b888-2248e3d9e981","Type":"ContainerDied","Data":"96952b99923f24adae5e9bce715087dafd62bee07d71fa923cba2aafdc421eb1"}
Oct 30 19:36:53.454053 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:53.454020    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-95d2ac-74dbfdbb99-fjbtf" event={"ID":"2133d6b7-b08a-46ac-b888-2248e3d9e981","Type":"ContainerDied","Data":"26279f832a8f0d05635c7ba1ca4aac29a96ad7788eae73386feb9302aebaa975"}
Oct 30 19:36:53.468997 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:53.468965    5043 scope.go:122] "RemoveContainer" containerID="fbd6b833037b34de08ecd362089deac03d9e2ae8b28ee1b5484d849860bb8b7a"
Oct 30 19:36:53.469478 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:36:53.469452    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbd6b833037b34de08ecd362089deac03d9e2ae8b28ee1b5484d849860bb8b7a\": container with ID starting with fbd6b833037b34de08ecd362089deac03d9e2ae8b28ee1b5484d849860bb8b7a not found: ID does not exist" containerID="fbd6b833037b34de08ecd362089deac03d9e2ae8b28ee1b5484d849860bb8b7a"
Oct 30 19:36:53.469583 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:53.469490    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbd6b833037b34de08ecd362089deac03d9e2ae8b28ee1b5484d849860bb8b7a"} err="failed to get container status \"fbd6b833037b34de08ecd362089deac03d9e2ae8b28ee1b5484d849860bb8b7a\": rpc error: code = NotFound desc = could not find container \"fbd6b833037b34de08ecd362089deac03d9e2ae8b28ee1b5484d849860bb8b7a\": container with ID starting with fbd6b833037b34de08ecd362089deac03d9e2ae8b28ee1b5484d849860bb8b7a not found: ID does not exist"
Oct 30 19:36:53.469583 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:53.469514    5043 scope.go:122] "RemoveContainer" containerID="96952b99923f24adae5e9bce715087dafd62bee07d71fa923cba2aafdc421eb1"
Oct 30 19:36:53.482497 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:53.482454    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-95d2ac-75f686bc4d-v7vw5"]
Oct 30 19:36:53.487186 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:53.487162    5043 scope.go:122] "RemoveContainer" containerID="96952b99923f24adae5e9bce715087dafd62bee07d71fa923cba2aafdc421eb1"
Oct 30 19:36:53.487500 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:36:53.487474    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96952b99923f24adae5e9bce715087dafd62bee07d71fa923cba2aafdc421eb1\": container with ID starting with 96952b99923f24adae5e9bce715087dafd62bee07d71fa923cba2aafdc421eb1 not found: ID does not exist" containerID="96952b99923f24adae5e9bce715087dafd62bee07d71fa923cba2aafdc421eb1"
Oct 30 19:36:53.487574 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:53.487512    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96952b99923f24adae5e9bce715087dafd62bee07d71fa923cba2aafdc421eb1"} err="failed to get container status \"96952b99923f24adae5e9bce715087dafd62bee07d71fa923cba2aafdc421eb1\": rpc error: code = NotFound desc = could not find container \"96952b99923f24adae5e9bce715087dafd62bee07d71fa923cba2aafdc421eb1\": container with ID starting with 96952b99923f24adae5e9bce715087dafd62bee07d71fa923cba2aafdc421eb1 not found: ID does not exist"
Oct 30 19:36:53.487846 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:53.487827    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-95d2ac-75f686bc4d-v7vw5"]
Oct 30 19:36:53.498678 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:53.498660    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-95d2ac-74dbfdbb99-fjbtf"]
Oct 30 19:36:53.505983 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:53.505962    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-95d2ac-74dbfdbb99-fjbtf"]
Oct 30 19:36:54.648073 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:54.648039    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2133d6b7-b08a-46ac-b888-2248e3d9e981" path="/var/lib/kubelet/pods/2133d6b7-b08a-46ac-b888-2248e3d9e981/volumes"
Oct 30 19:36:54.648541 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:54.648527    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e5c13a6-0354-4d98-991a-ceb6ea271fb1" path="/var/lib/kubelet/pods/6e5c13a6-0354-4d98-991a-ceb6ea271fb1/volumes"
Oct 30 19:36:54.980313 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:36:54.980280    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:36:57.370157 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:57.370122    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e66fa4-f72p2"]
Oct 30 19:36:57.383365 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:57.383326    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e66fa4-f72p2"]
Oct 30 19:36:58.740771 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:36:58.737908    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60621b88-baeb-4d41-b665-7489d0c8ff7c" path="/var/lib/kubelet/pods/60621b88-baeb-4d41-b665-7489d0c8ff7c/volumes"
Oct 30 19:37:05.615763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:05.615151    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:05.764758 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:05.762167    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:05.893816 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:05.892818    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:37:05.929871 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:05.929825    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:05.950758 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:05.943871    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:06.205811 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:06.201887    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:06.220044 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:06.214911    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:06.567762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:06.552001    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:06.640761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:06.623842    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:06.640761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:06.624444    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:37:06.809500 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:06.808349    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:37:06.937823 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:06.896492    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:07.167507 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:06.940483    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:37:07.168244 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:06.972198    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:07.168628 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:06.981853    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": context deadline exceeded"
Oct 30 19:37:07.168779 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:07.138006    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:07.221867 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:07.197479    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:37:07.221867 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:07.197609    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:37:07.221867 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:07.197643    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:37:07.221867 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:07.197676    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"
Oct 30 19:37:07.229772 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:07.223120    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:07.229772 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:07.223680    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:37:07.229772 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:07.224479    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:07.245052 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:07.245025    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:37:07.712856 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:07.691694    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:37:08.158231 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:07.716772    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:37:08.160412 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:07.718388    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": context deadline exceeded"
Oct 30 19:37:08.160673 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:07.844138    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:08.167044 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:08.167010    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:37:08.195778 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:07.936223    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": context deadline exceeded"
Oct 30 19:37:08.195778 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:07.846023    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 30 19:37:08.241376 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:08.241317    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:37:08.290830 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:08.247837    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:37:08.430412 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:08.248005    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:37:08.458761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:08.248133    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded"
Oct 30 19:37:08.508066 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:08.248307    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:37:08.510090 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:08.301428    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:37:08.517946 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:08.517915    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:37:09.165767 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:09.164849    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:09.174856 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:09.174070    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:37:09.215804 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:09.215762    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:09.244905 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:09.244858    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:09.387858 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:09.358452    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:09.387858 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:09.363090    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded"
Oct 30 19:37:09.401776 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:09.394112    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:09.512980 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:09.505876    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:09.512980 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:09.510712    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:09.512980 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:09.510821    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded"
Oct 30 19:37:09.512980 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:09.510867    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:37:10.158244 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.158204    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": context deadline exceeded"
Oct 30 19:37:10.238556 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.238513    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zt5b2" podUID="a348d1da-3b22-4650-84a5-4e2de27d37e3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:37:10.258918 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.258797    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:10.353756 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.331335    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:10.523799 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:37:10.513607    5043 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.831s"
Oct 30 19:37:10.523799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.513701    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"
Oct 30 19:37:10.523799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.513770    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"
Oct 30 19:37:10.523799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.513862    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:37:10.523799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.513902    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:37:10.523799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.513932    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:37:10.523799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.513960    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:37:10.523799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.513984    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-73f96d-szmhm"]
Oct 30 19:37:10.523799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.515073    5043 cpu_manager.go:414] "containerMap: removing container" podUID="6e5c13a6-0354-4d98-991a-ceb6ea271fb1" containerName="payload-processing"
Oct 30 19:37:10.523799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.515094    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e5c13a6-0354-4d98-991a-ceb6ea271fb1" containerName="payload-processing"
Oct 30 19:37:10.523799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.515118    5043 cpu_manager.go:414] "containerMap: removing container" podUID="143f7a39-3eed-4dd0-9ef9-dc06ea5c4e4b" containerName="payload-processing"
Oct 30 19:37:10.523799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.515126    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="143f7a39-3eed-4dd0-9ef9-dc06ea5c4e4b" containerName="payload-processing"
Oct 30 19:37:10.523799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.515144    5043 cpu_manager.go:414] "containerMap: removing container" podUID="fe77a5ed-7a32-4350-b60c-d119d766eb1c" containerName="payload-pre-processing"
Oct 30 19:37:10.523799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.515153    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe77a5ed-7a32-4350-b60c-d119d766eb1c" containerName="payload-pre-processing"
Oct 30 19:37:10.523799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.515180    5043 cpu_manager.go:414] "containerMap: removing container" podUID="2133d6b7-b08a-46ac-b888-2248e3d9e981" containerName="payload-pre-processing"
Oct 30 19:37:10.523799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.515188    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2133d6b7-b08a-46ac-b888-2248e3d9e981" containerName="payload-pre-processing"
Oct 30 19:37:10.523799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.515223    5043 cpu_manager.go:414] "containerMap: removing container" podUID="0aa81e41-151e-47b9-8a09-f4cf20a52dad" containerName="istio-proxy"
Oct 30 19:37:10.523799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.515230    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aa81e41-151e-47b9-8a09-f4cf20a52dad" containerName="istio-proxy"
Oct 30 19:37:10.523799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.515245    5043 cpu_manager.go:414] "containerMap: removing container" podUID="34027f03-ad24-4f28-b53d-8e21422dac2e" containerName="revoke-keys"
Oct 30 19:37:10.523799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.515251    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34027f03-ad24-4f28-b53d-8e21422dac2e" containerName="revoke-keys"
Oct 30 19:37:10.523799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.515262    5043 cpu_manager.go:414] "containerMap: removing container" podUID="03c6df78-8285-46c7-b5bd-896a0350c1de" containerName="istio-proxy"
Oct 30 19:37:10.523799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.515268    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03c6df78-8285-46c7-b5bd-896a0350c1de" containerName="istio-proxy"
Oct 30 19:37:10.523799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.515431    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="143f7a39-3eed-4dd0-9ef9-dc06ea5c4e4b" containerName="payload-processing"
Oct 30 19:37:10.523799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.515446    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe77a5ed-7a32-4350-b60c-d119d766eb1c" containerName="payload-pre-processing"
Oct 30 19:37:10.523799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.515459    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="2133d6b7-b08a-46ac-b888-2248e3d9e981" containerName="payload-pre-processing"
Oct 30 19:37:10.523799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.515473    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aa81e41-151e-47b9-8a09-f4cf20a52dad" containerName="istio-proxy"
Oct 30 19:37:10.523799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.515484    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="34027f03-ad24-4f28-b53d-8e21422dac2e" containerName="revoke-keys"
Oct 30 19:37:10.523799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.515494    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="03c6df78-8285-46c7-b5bd-896a0350c1de" containerName="istio-proxy"
Oct 30 19:37:10.523799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.515508    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e5c13a6-0354-4d98-991a-ceb6ea271fb1" containerName="payload-processing"
Oct 30 19:37:10.654926 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.654895    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-73f96d-szmhm"
Oct 30 19:37:10.689764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.684049    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:37:10.689764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.684118    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:37:10.689764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.684207    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:37:10.752915 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.752881    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:37:10.753137 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.752945    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-73f96d-szmhm"]
Oct 30 19:37:10.753137 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.753028    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5294a7b-62g49"]
Oct 30 19:37:10.753137 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.753080    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"
Oct 30 19:37:10.753137 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.753101    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5294a7b-62g49"]
Oct 30 19:37:10.753355 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.753166    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:37:10.753355 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.753207    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:37:10.753355 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.753243    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:37:10.827827 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.769596    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:37:10.874457 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.874413    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxv8h\" (UniqueName: \"kubernetes.io/projected/1a7c6064-437f-46b9-93bd-a9b2cb31ede3-kube-api-access-qxv8h\") pod \"maas-api-revoke-keys-e2e-api-a-73f96d-szmhm\" (UID: \"1a7c6064-437f-46b9-93bd-a9b2cb31ede3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-73f96d-szmhm"
Oct 30 19:37:10.890513 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.890097    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a7c6064-437f-46b9-93bd-a9b2cb31ede3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-73f96d-szmhm\" (UID: \"1a7c6064-437f-46b9-93bd-a9b2cb31ede3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-73f96d-szmhm"
Oct 30 19:37:10.891871 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:10.891844    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a7c6064-437f-46b9-93bd-a9b2cb31ede3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-73f96d-szmhm\" (UID: \"1a7c6064-437f-46b9-93bd-a9b2cb31ede3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-73f96d-szmhm"
Oct 30 19:37:11.066772 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:11.012220    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a7c6064-437f-46b9-93bd-a9b2cb31ede3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-73f96d-szmhm\" (UID: \"1a7c6064-437f-46b9-93bd-a9b2cb31ede3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-73f96d-szmhm"
Oct 30 19:37:11.068457 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:11.068419    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxv8h\" (UniqueName: \"kubernetes.io/projected/1a7c6064-437f-46b9-93bd-a9b2cb31ede3-kube-api-access-qxv8h\") pod \"maas-api-revoke-keys-e2e-api-a-73f96d-szmhm\" (UID: \"1a7c6064-437f-46b9-93bd-a9b2cb31ede3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-73f96d-szmhm"
Oct 30 19:37:11.068693 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:11.068675    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a7c6064-437f-46b9-93bd-a9b2cb31ede3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-73f96d-szmhm\" (UID: \"1a7c6064-437f-46b9-93bd-a9b2cb31ede3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-73f96d-szmhm"
Oct 30 19:37:11.129939 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:11.110253    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a7c6064-437f-46b9-93bd-a9b2cb31ede3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-73f96d-szmhm\" (UID: \"1a7c6064-437f-46b9-93bd-a9b2cb31ede3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-73f96d-szmhm"
Oct 30 19:37:11.152829 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:11.150845    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a7c6064-437f-46b9-93bd-a9b2cb31ede3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-73f96d-szmhm\" (UID: \"1a7c6064-437f-46b9-93bd-a9b2cb31ede3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-73f96d-szmhm"
Oct 30 19:37:11.422816 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:11.419105    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxv8h\" (UniqueName: \"kubernetes.io/projected/1a7c6064-437f-46b9-93bd-a9b2cb31ede3-kube-api-access-qxv8h\") pod \"maas-api-revoke-keys-e2e-api-a-73f96d-szmhm\" (UID: \"1a7c6064-437f-46b9-93bd-a9b2cb31ede3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-73f96d-szmhm"
Oct 30 19:37:11.738173 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:11.738141    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-73f96d-szmhm"
Oct 30 19:37:12.178771 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:12.178627    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:12.244760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:12.209926    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:37:12.655458 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:12.655420    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23d0b40e-410b-4319-bad7-d84046f36191" path="/var/lib/kubelet/pods/23d0b40e-410b-4319-bad7-d84046f36191/volumes"
Oct 30 19:37:12.723001 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:12.721088    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-73f96d-szmhm"]
Oct 30 19:37:12.724600 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:37:12.724550    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a7c6064_437f_46b9_93bd_a9b2cb31ede3.slice/crio-5a0cd9771b39b8075647b8fb4a464ad08f5b0ed15e56f0bc552d17c26ca58f19 WatchSource:0}: Error finding container 5a0cd9771b39b8075647b8fb4a464ad08f5b0ed15e56f0bc552d17c26ca58f19: Status 404 returned error can't find the container with id 5a0cd9771b39b8075647b8fb4a464ad08f5b0ed15e56f0bc552d17c26ca58f19
Oct 30 19:37:12.840304 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:12.840123    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c"]
Oct 30 19:37:12.840489 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:12.840400    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c" podUID="33a5f4d9-fe3f-40ad-904d-db5ac720c81e" containerName="istio-proxy" containerID="cri-o://43dd5570ad53ac67d91af92c664436a5e37afb084da44e24a09abb05a57bfb81" gracePeriod=30
Oct 30 19:37:13.485762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:13.482917    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-73f96d-szmhm" event={"ID":"1a7c6064-437f-46b9-93bd-a9b2cb31ede3","Type":"ContainerStarted","Data":"da655624aeff9d702a50182e5e2dcbf4a7f888b465ae83e3b439fa5c0216db25"}
Oct 30 19:37:13.485762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:13.482985    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-73f96d-szmhm" event={"ID":"1a7c6064-437f-46b9-93bd-a9b2cb31ede3","Type":"ContainerStarted","Data":"5a0cd9771b39b8075647b8fb4a464ad08f5b0ed15e56f0bc552d17c26ca58f19"}
Oct 30 19:37:13.509772 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:13.505146    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-73f96d-szmhm" podStartSLOduration=4.505123524 podStartE2EDuration="4.505123524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:13.502081265 +0000 UTC m=+1253.013141230" watchObservedRunningTime="2026-10-30 19:37:13.505123524 +0000 UTC m=+1253.016183490"
Oct 30 19:37:14.498445 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:14.498405    5043 generic.go:356] "Generic (PLEG): container finished" podID="1a7c6064-437f-46b9-93bd-a9b2cb31ede3" containerID="da655624aeff9d702a50182e5e2dcbf4a7f888b465ae83e3b439fa5c0216db25" exitCode=0
Oct 30 19:37:14.499006 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:14.498504    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-73f96d-szmhm" event={"ID":"1a7c6064-437f-46b9-93bd-a9b2cb31ede3","Type":"ContainerDied","Data":"da655624aeff9d702a50182e5e2dcbf4a7f888b465ae83e3b439fa5c0216db25"}
Oct 30 19:37:15.199661 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:15.199617    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-82012b-5987f89c59-xqtgs"]
Oct 30 19:37:15.206676 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:15.206597    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-82012b-5987f89c59-xqtgs"
Oct 30 19:37:15.217147 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:15.217042    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-82012b-dockercfg-sx2mn\""
Oct 30 19:37:15.217147 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:15.217071    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-82012b-5987f89c59-xqtgs"]
Oct 30 19:37:15.217147 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:15.217114    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-82012b\""
Oct 30 19:37:15.244169 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:15.244138    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jng5\" (UniqueName: \"kubernetes.io/projected/d41c7166-ac32-4778-8af3-3d32819427e3-kube-api-access-7jng5\") pod \"payload-pre-processing-e2e-api-a-82012b-5987f89c59-xqtgs\" (UID: \"d41c7166-ac32-4778-8af3-3d32819427e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-82012b-5987f89c59-xqtgs"
Oct 30 19:37:15.244423 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:15.244399    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d41c7166-ac32-4778-8af3-3d32819427e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-82012b-5987f89c59-xqtgs\" (UID: \"d41c7166-ac32-4778-8af3-3d32819427e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-82012b-5987f89c59-xqtgs"
Oct 30 19:37:15.611494 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:15.392535    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jng5\" (UniqueName: \"kubernetes.io/projected/d41c7166-ac32-4778-8af3-3d32819427e3-kube-api-access-7jng5\") pod \"payload-pre-processing-e2e-api-a-82012b-5987f89c59-xqtgs\" (UID: \"d41c7166-ac32-4778-8af3-3d32819427e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-82012b-5987f89c59-xqtgs"
Oct 30 19:37:15.612604 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:15.567229    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jng5\" (UniqueName: \"kubernetes.io/projected/d41c7166-ac32-4778-8af3-3d32819427e3-kube-api-access-7jng5\") pod \"payload-pre-processing-e2e-api-a-82012b-5987f89c59-xqtgs\" (UID: \"d41c7166-ac32-4778-8af3-3d32819427e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-82012b-5987f89c59-xqtgs"
Oct 30 19:37:15.612890 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:15.612780    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d41c7166-ac32-4778-8af3-3d32819427e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-82012b-5987f89c59-xqtgs\" (UID: \"d41c7166-ac32-4778-8af3-3d32819427e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-82012b-5987f89c59-xqtgs"
Oct 30 19:37:15.622294 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:15.614447    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d41c7166-ac32-4778-8af3-3d32819427e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-82012b-5987f89c59-xqtgs\" (UID: \"d41c7166-ac32-4778-8af3-3d32819427e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-82012b-5987f89c59-xqtgs"
Oct 30 19:37:15.848813 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:15.847575    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-82012b-5987f89c59-xqtgs"
Oct 30 19:37:16.037375 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:16.037345    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-73f96d-szmhm"
Oct 30 19:37:16.211451 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:16.211412    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a7c6064-437f-46b9-93bd-a9b2cb31ede3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a7c6064-437f-46b9-93bd-a9b2cb31ede3-maas-api-xks-ca\") pod \"1a7c6064-437f-46b9-93bd-a9b2cb31ede3\" (UID: \"1a7c6064-437f-46b9-93bd-a9b2cb31ede3\") "
Oct 30 19:37:16.221764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:16.211781    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a7c6064-437f-46b9-93bd-a9b2cb31ede3-kube-api-access-qxv8h\" (UniqueName: \"kubernetes.io/projected/1a7c6064-437f-46b9-93bd-a9b2cb31ede3-kube-api-access-qxv8h\") pod \"1a7c6064-437f-46b9-93bd-a9b2cb31ede3\" (UID: \"1a7c6064-437f-46b9-93bd-a9b2cb31ede3\") "
Oct 30 19:37:16.222326 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:16.222300    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a7c6064-437f-46b9-93bd-a9b2cb31ede3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a7c6064-437f-46b9-93bd-a9b2cb31ede3-maas-api-openshift-ca\") pod \"1a7c6064-437f-46b9-93bd-a9b2cb31ede3\" (UID: \"1a7c6064-437f-46b9-93bd-a9b2cb31ede3\") "
Oct 30 19:37:16.240595 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:16.229891    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a7c6064-437f-46b9-93bd-a9b2cb31ede3-maas-api-openshift-ca" pod "1a7c6064-437f-46b9-93bd-a9b2cb31ede3" (UID: "1a7c6064-437f-46b9-93bd-a9b2cb31ede3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:37:16.240595 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:16.232824    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a7c6064-437f-46b9-93bd-a9b2cb31ede3-kube-api-access-qxv8h" pod "1a7c6064-437f-46b9-93bd-a9b2cb31ede3" (UID: "1a7c6064-437f-46b9-93bd-a9b2cb31ede3"). InnerVolumeSpecName "kube-api-access-qxv8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:16.251812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:16.248784    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a7c6064-437f-46b9-93bd-a9b2cb31ede3-maas-api-xks-ca" pod "1a7c6064-437f-46b9-93bd-a9b2cb31ede3" (UID: "1a7c6064-437f-46b9-93bd-a9b2cb31ede3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:37:16.264510 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:16.262278    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-82012b-5987f89c59-xqtgs"]
Oct 30 19:37:16.279817 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:37:16.272920    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd41c7166_ac32_4778_8af3_3d32819427e3.slice/crio-7deef02642f4c1d933653e90d1ec0bb2656764db4b703e604b6fc19ca852ff98 WatchSource:0}: Error finding container 7deef02642f4c1d933653e90d1ec0bb2656764db4b703e604b6fc19ca852ff98: Status 404 returned error can't find the container with id 7deef02642f4c1d933653e90d1ec0bb2656764db4b703e604b6fc19ca852ff98
Oct 30 19:37:16.325619 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:16.325582    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a7c6064-437f-46b9-93bd-a9b2cb31ede3-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:16.325775 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:16.325623    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxv8h\" (UniqueName: \"kubernetes.io/projected/1a7c6064-437f-46b9-93bd-a9b2cb31ede3-kube-api-access-qxv8h\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:16.325775 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:16.325640    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a7c6064-437f-46b9-93bd-a9b2cb31ede3-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:16.591234 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:16.591155    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-73f96d-szmhm"
Oct 30 19:37:16.591234 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:16.591199    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-73f96d-szmhm" event={"ID":"1a7c6064-437f-46b9-93bd-a9b2cb31ede3","Type":"ContainerDied","Data":"5a0cd9771b39b8075647b8fb4a464ad08f5b0ed15e56f0bc552d17c26ca58f19"}
Oct 30 19:37:16.591503 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:16.591256    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a0cd9771b39b8075647b8fb4a464ad08f5b0ed15e56f0bc552d17c26ca58f19"
Oct 30 19:37:16.593711 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:16.593677    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-82012b-5987f89c59-xqtgs" event={"ID":"d41c7166-ac32-4778-8af3-3d32819427e3","Type":"ContainerStarted","Data":"f365854e03ba3afe0c420f7d98e2e5d8ecb4a46404ddb5cac5016a8597c939e1"}
Oct 30 19:37:16.593924 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:16.593752    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-82012b-5987f89c59-xqtgs" event={"ID":"d41c7166-ac32-4778-8af3-3d32819427e3","Type":"ContainerStarted","Data":"7deef02642f4c1d933653e90d1ec0bb2656764db4b703e604b6fc19ca852ff98"}
Oct 30 19:37:16.593924 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:16.593820    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-82012b-5987f89c59-xqtgs"
Oct 30 19:37:16.616290 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:16.616219    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-82012b-5987f89c59-xqtgs" podStartSLOduration=1.6161992299999999 podStartE2EDuration="1.61619923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:16.613100205 +0000 UTC m=+1256.124160168" watchObservedRunningTime="2026-10-30 19:37:16.61619923 +0000 UTC m=+1256.127259195"
Oct 30 19:37:17.253806 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:17.253762    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-89a3f6-db46ffd87-8l4s5"]
Oct 30 19:37:17.254608 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:17.254593    5043 cpu_manager.go:414] "containerMap: removing container" podUID="1a7c6064-437f-46b9-93bd-a9b2cb31ede3" containerName="revoke-keys"
Oct 30 19:37:17.254652 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:17.254610    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a7c6064-437f-46b9-93bd-a9b2cb31ede3" containerName="revoke-keys"
Oct 30 19:37:17.254770 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:17.254759    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a7c6064-437f-46b9-93bd-a9b2cb31ede3" containerName="revoke-keys"
Oct 30 19:37:17.296245 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:17.295012    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-89a3f6-db46ffd87-8l4s5"]
Oct 30 19:37:17.296245 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:17.295174    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89a3f6-db46ffd87-8l4s5"
Oct 30 19:37:17.306762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:17.302085    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-89a3f6-dockercfg-lngqf\""
Oct 30 19:37:17.306762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:17.303505    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-89a3f6\""
Oct 30 19:37:17.526831 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:17.499679    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62c3de5f-9601-43f4-a89a-85c623e1222b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-89a3f6-db46ffd87-8l4s5\" (UID: \"62c3de5f-9601-43f4-a89a-85c623e1222b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89a3f6-db46ffd87-8l4s5"
Oct 30 19:37:17.526831 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:17.499889    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-869j4\" (UniqueName: \"kubernetes.io/projected/62c3de5f-9601-43f4-a89a-85c623e1222b-kube-api-access-869j4\") pod \"payload-pre-processing-e2e-inf-a-89a3f6-db46ffd87-8l4s5\" (UID: \"62c3de5f-9601-43f4-a89a-85c623e1222b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89a3f6-db46ffd87-8l4s5"
Oct 30 19:37:17.624760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:17.624363    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62c3de5f-9601-43f4-a89a-85c623e1222b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-89a3f6-db46ffd87-8l4s5\" (UID: \"62c3de5f-9601-43f4-a89a-85c623e1222b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89a3f6-db46ffd87-8l4s5"
Oct 30 19:37:17.626930 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:17.626866    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62c3de5f-9601-43f4-a89a-85c623e1222b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-89a3f6-db46ffd87-8l4s5\" (UID: \"62c3de5f-9601-43f4-a89a-85c623e1222b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89a3f6-db46ffd87-8l4s5"
Oct 30 19:37:17.627345 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:17.627319    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-869j4\" (UniqueName: \"kubernetes.io/projected/62c3de5f-9601-43f4-a89a-85c623e1222b-kube-api-access-869j4\") pod \"payload-pre-processing-e2e-inf-a-89a3f6-db46ffd87-8l4s5\" (UID: \"62c3de5f-9601-43f4-a89a-85c623e1222b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89a3f6-db46ffd87-8l4s5"
Oct 30 19:37:17.683388 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:17.683359    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-869j4\" (UniqueName: \"kubernetes.io/projected/62c3de5f-9601-43f4-a89a-85c623e1222b-kube-api-access-869j4\") pod \"payload-pre-processing-e2e-inf-a-89a3f6-db46ffd87-8l4s5\" (UID: \"62c3de5f-9601-43f4-a89a-85c623e1222b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89a3f6-db46ffd87-8l4s5"
Oct 30 19:37:17.937690 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:17.937662    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89a3f6-db46ffd87-8l4s5"
Oct 30 19:37:18.304838 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:18.296283    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-89a3f6-db46ffd87-8l4s5"]
Oct 30 19:37:18.304838 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:18.298922    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c"
Oct 30 19:37:18.305349 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:37:18.305318    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62c3de5f_9601_43f4_a89a_85c623e1222b.slice/crio-c55e1ce7e0548eba77d04bbe4d83caaaff86831cd936c8e4d1fe2182677814b6 WatchSource:0}: Error finding container c55e1ce7e0548eba77d04bbe4d83caaaff86831cd936c8e4d1fe2182677814b6: Status 404 returned error can't find the container with id c55e1ce7e0548eba77d04bbe4d83caaaff86831cd936c8e4d1fe2182677814b6
Oct 30 19:37:18.352504 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:18.352475    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-istio-data\") pod \"33a5f4d9-fe3f-40ad-904d-db5ac720c81e\" (UID: \"33a5f4d9-fe3f-40ad-904d-db5ac720c81e\") "
Oct 30 19:37:18.352721 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:18.352574    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-istiod-ca-cert\") pod \"33a5f4d9-fe3f-40ad-904d-db5ac720c81e\" (UID: \"33a5f4d9-fe3f-40ad-904d-db5ac720c81e\") "
Oct 30 19:37:18.352721 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:18.352664    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-istio-envoy\") pod \"33a5f4d9-fe3f-40ad-904d-db5ac720c81e\" (UID: \"33a5f4d9-fe3f-40ad-904d-db5ac720c81e\") "
Oct 30 19:37:18.352721 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:18.352714    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-workload-certs\") pod \"33a5f4d9-fe3f-40ad-904d-db5ac720c81e\" (UID: \"33a5f4d9-fe3f-40ad-904d-db5ac720c81e\") "
Oct 30 19:37:18.352955 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:18.352780    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-workload-socket\") pod \"33a5f4d9-fe3f-40ad-904d-db5ac720c81e\" (UID: \"33a5f4d9-fe3f-40ad-904d-db5ac720c81e\") "
Oct 30 19:37:18.352955 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:18.352779    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-istio-data" pod "33a5f4d9-fe3f-40ad-904d-db5ac720c81e" (UID: "33a5f4d9-fe3f-40ad-904d-db5ac720c81e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:18.352955 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:18.352851    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-istio-token\" (UniqueName: \"kubernetes.io/projected/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-istio-token\") pod \"33a5f4d9-fe3f-40ad-904d-db5ac720c81e\" (UID: \"33a5f4d9-fe3f-40ad-904d-db5ac720c81e\") "
Oct 30 19:37:18.352955 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:18.352949    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-credential-socket\") pod \"33a5f4d9-fe3f-40ad-904d-db5ac720c81e\" (UID: \"33a5f4d9-fe3f-40ad-904d-db5ac720c81e\") "
Oct 30 19:37:18.353153 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:18.353021    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-istio-podinfo\") pod \"33a5f4d9-fe3f-40ad-904d-db5ac720c81e\" (UID: \"33a5f4d9-fe3f-40ad-904d-db5ac720c81e\") "
Oct 30 19:37:18.353323 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:18.353239    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-kube-api-access-8nhhh\" (UniqueName: \"kubernetes.io/projected/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-kube-api-access-8nhhh\") pod \"33a5f4d9-fe3f-40ad-904d-db5ac720c81e\" (UID: \"33a5f4d9-fe3f-40ad-904d-db5ac720c81e\") "
Oct 30 19:37:18.353432 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:18.353322    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-workload-socket" pod "33a5f4d9-fe3f-40ad-904d-db5ac720c81e" (UID: "33a5f4d9-fe3f-40ad-904d-db5ac720c81e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:18.353531 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:18.353505    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-workload-certs" pod "33a5f4d9-fe3f-40ad-904d-db5ac720c81e" (UID: "33a5f4d9-fe3f-40ad-904d-db5ac720c81e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:18.353585 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:18.353548    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-istiod-ca-cert" pod "33a5f4d9-fe3f-40ad-904d-db5ac720c81e" (UID: "33a5f4d9-fe3f-40ad-904d-db5ac720c81e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:37:18.354118 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:18.354055    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-credential-socket" pod "33a5f4d9-fe3f-40ad-904d-db5ac720c81e" (UID: "33a5f4d9-fe3f-40ad-904d-db5ac720c81e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:18.355873 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:18.354392    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:18.355873 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:18.354423    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:18.355873 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:18.354439    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:18.355873 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:18.354496    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:18.355873 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:18.354511    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:18.362173 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:18.362088    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-kube-api-access-8nhhh" pod "33a5f4d9-fe3f-40ad-904d-db5ac720c81e" (UID: "33a5f4d9-fe3f-40ad-904d-db5ac720c81e"). InnerVolumeSpecName "kube-api-access-8nhhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:18.367804 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:18.367763    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-istio-podinfo" pod "33a5f4d9-fe3f-40ad-904d-db5ac720c81e" (UID: "33a5f4d9-fe3f-40ad-904d-db5ac720c81e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:37:18.368178 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:18.368152    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-istio-envoy" pod "33a5f4d9-fe3f-40ad-904d-db5ac720c81e" (UID: "33a5f4d9-fe3f-40ad-904d-db5ac720c81e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:18.383384 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:18.383345    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-istio-token" pod "33a5f4d9-fe3f-40ad-904d-db5ac720c81e" (UID: "33a5f4d9-fe3f-40ad-904d-db5ac720c81e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:18.456426 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:18.456380    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:18.456426 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:18.456427    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:18.456718 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:18.456443    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:18.456718 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:18.456460    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nhhh\" (UniqueName: \"kubernetes.io/projected/33a5f4d9-fe3f-40ad-904d-db5ac720c81e-kube-api-access-8nhhh\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:18.651914 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:18.651837    5043 generic.go:356] "Generic (PLEG): container finished" podID="33a5f4d9-fe3f-40ad-904d-db5ac720c81e" containerID="43dd5570ad53ac67d91af92c664436a5e37afb084da44e24a09abb05a57bfb81" exitCode=0
Oct 30 19:37:18.654601 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:18.651911    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c"
Oct 30 19:37:18.654601 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:18.651920    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c" event={"ID":"33a5f4d9-fe3f-40ad-904d-db5ac720c81e","Type":"ContainerDied","Data":"43dd5570ad53ac67d91af92c664436a5e37afb084da44e24a09abb05a57bfb81"}
Oct 30 19:37:18.654601 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:18.651969    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c" event={"ID":"33a5f4d9-fe3f-40ad-904d-db5ac720c81e","Type":"ContainerDied","Data":"5b450fd6973a5e7c1332bd50bf684b29350cca0d1005082c2050f06333a53118"}
Oct 30 19:37:18.654601 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:18.651993    5043 scope.go:122] "RemoveContainer" containerID="43dd5570ad53ac67d91af92c664436a5e37afb084da44e24a09abb05a57bfb81"
Oct 30 19:37:18.654601 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:18.654384    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89a3f6-db46ffd87-8l4s5" event={"ID":"62c3de5f-9601-43f4-a89a-85c623e1222b","Type":"ContainerStarted","Data":"7ad16887948a82336cccb54ea5544800808823a9636386c02057bbaa8ac5316f"}
Oct 30 19:37:18.654601 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:18.654431    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89a3f6-db46ffd87-8l4s5" event={"ID":"62c3de5f-9601-43f4-a89a-85c623e1222b","Type":"ContainerStarted","Data":"c55e1ce7e0548eba77d04bbe4d83caaaff86831cd936c8e4d1fe2182677814b6"}
Oct 30 19:37:18.655164 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:18.654962    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89a3f6-db46ffd87-8l4s5"
Oct 30 19:37:18.673906 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:18.673857    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89a3f6-db46ffd87-8l4s5" podStartSLOduration=1.673839614 podStartE2EDuration="1.673839614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:18.67168416 +0000 UTC m=+1258.182744134" watchObservedRunningTime="2026-10-30 19:37:18.673839614 +0000 UTC m=+1258.184899580"
Oct 30 19:37:18.700013 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:18.699987    5043 scope.go:122] "RemoveContainer" containerID="43dd5570ad53ac67d91af92c664436a5e37afb084da44e24a09abb05a57bfb81"
Oct 30 19:37:18.700419 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:37:18.700387    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43dd5570ad53ac67d91af92c664436a5e37afb084da44e24a09abb05a57bfb81\": container with ID starting with 43dd5570ad53ac67d91af92c664436a5e37afb084da44e24a09abb05a57bfb81 not found: ID does not exist" containerID="43dd5570ad53ac67d91af92c664436a5e37afb084da44e24a09abb05a57bfb81"
Oct 30 19:37:18.700617 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:18.700576    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43dd5570ad53ac67d91af92c664436a5e37afb084da44e24a09abb05a57bfb81"} err="failed to get container status \"43dd5570ad53ac67d91af92c664436a5e37afb084da44e24a09abb05a57bfb81\": rpc error: code = NotFound desc = could not find container \"43dd5570ad53ac67d91af92c664436a5e37afb084da44e24a09abb05a57bfb81\": container with ID starting with 43dd5570ad53ac67d91af92c664436a5e37afb084da44e24a09abb05a57bfb81 not found: ID does not exist"
Oct 30 19:37:18.725080 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:18.725055    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c"]
Oct 30 19:37:18.729713 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:18.729694    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-nkc9c"]
Oct 30 19:37:19.403481 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:19.403448    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-73f96d-65c48fbd9f-98p49"]
Oct 30 19:37:19.528369 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:19.528339    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-60577a-fg75m"]
Oct 30 19:37:19.529229 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:19.529213    5043 cpu_manager.go:414] "containerMap: removing container" podUID="33a5f4d9-fe3f-40ad-904d-db5ac720c81e" containerName="istio-proxy"
Oct 30 19:37:19.529286 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:19.529230    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33a5f4d9-fe3f-40ad-904d-db5ac720c81e" containerName="istio-proxy"
Oct 30 19:37:19.529366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:19.529357    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="33a5f4d9-fe3f-40ad-904d-db5ac720c81e" containerName="istio-proxy"
Oct 30 19:37:19.533212 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:19.533194    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-60577a-fg75m"
Oct 30 19:37:19.544888 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:19.544864    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:37:19.545140 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:19.544875    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:37:19.546307 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:19.546285    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:37:19.546408 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:19.546373    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:37:19.551374 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:19.551349    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-60577a-fg75m"]
Oct 30 19:37:19.580577 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:19.580543    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4mgj\" (UniqueName: \"kubernetes.io/projected/88bba8ee-11c4-48f7-80c8-d3a636ea8a1c-kube-api-access-d4mgj\") pod \"maas-api-revoke-keys-e2e-worker-w5-60577a-fg75m\" (UID: \"88bba8ee-11c4-48f7-80c8-d3a636ea8a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-60577a-fg75m"
Oct 30 19:37:19.581146 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:19.580843    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88bba8ee-11c4-48f7-80c8-d3a636ea8a1c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-60577a-fg75m\" (UID: \"88bba8ee-11c4-48f7-80c8-d3a636ea8a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-60577a-fg75m"
Oct 30 19:37:19.581146 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:19.580941    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88bba8ee-11c4-48f7-80c8-d3a636ea8a1c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-60577a-fg75m\" (UID: \"88bba8ee-11c4-48f7-80c8-d3a636ea8a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-60577a-fg75m"
Oct 30 19:37:19.666990 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:19.666905    5043 generic.go:356] "Generic (PLEG): container finished" podID="aa769f06-2fc2-40ce-bb1a-a198b2274afb" containerID="84556bc09334bd7c38da4f9ec3c9ba09f614085869b5048a421d05117f8210de" exitCode=0
Oct 30 19:37:19.666990 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:19.666975    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5" event={"ID":"aa769f06-2fc2-40ce-bb1a-a198b2274afb","Type":"ContainerDied","Data":"84556bc09334bd7c38da4f9ec3c9ba09f614085869b5048a421d05117f8210de"}
Oct 30 19:37:19.667558 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:19.667538    5043 scope.go:122] "RemoveContainer" containerID="84556bc09334bd7c38da4f9ec3c9ba09f614085869b5048a421d05117f8210de"
Oct 30 19:37:19.669725 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:19.669697    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:37:19.669843 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:19.669792    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:37:19.669843 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:19.669828    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:37:19.682334 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:19.682308    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4mgj\" (UniqueName: \"kubernetes.io/projected/88bba8ee-11c4-48f7-80c8-d3a636ea8a1c-kube-api-access-d4mgj\") pod \"maas-api-revoke-keys-e2e-worker-w5-60577a-fg75m\" (UID: \"88bba8ee-11c4-48f7-80c8-d3a636ea8a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-60577a-fg75m"
Oct 30 19:37:19.682498 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:19.682385    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88bba8ee-11c4-48f7-80c8-d3a636ea8a1c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-60577a-fg75m\" (UID: \"88bba8ee-11c4-48f7-80c8-d3a636ea8a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-60577a-fg75m"
Oct 30 19:37:19.682498 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:19.682465    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88bba8ee-11c4-48f7-80c8-d3a636ea8a1c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-60577a-fg75m\" (UID: \"88bba8ee-11c4-48f7-80c8-d3a636ea8a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-60577a-fg75m"
Oct 30 19:37:19.683283 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:19.683258    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88bba8ee-11c4-48f7-80c8-d3a636ea8a1c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-60577a-fg75m\" (UID: \"88bba8ee-11c4-48f7-80c8-d3a636ea8a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-60577a-fg75m"
Oct 30 19:37:19.685219 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:19.685191    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88bba8ee-11c4-48f7-80c8-d3a636ea8a1c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-60577a-fg75m\" (UID: \"88bba8ee-11c4-48f7-80c8-d3a636ea8a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-60577a-fg75m"
Oct 30 19:37:19.697999 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:19.697971    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4mgj\" (UniqueName: \"kubernetes.io/projected/88bba8ee-11c4-48f7-80c8-d3a636ea8a1c-kube-api-access-d4mgj\") pod \"maas-api-revoke-keys-e2e-worker-w5-60577a-fg75m\" (UID: \"88bba8ee-11c4-48f7-80c8-d3a636ea8a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-60577a-fg75m"
Oct 30 19:37:19.855849 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:19.854113    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-60577a-fg75m"
Oct 30 19:37:20.491459 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:20.491422    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-60577a-fg75m"]
Oct 30 19:37:20.742138 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:20.742098    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33a5f4d9-fe3f-40ad-904d-db5ac720c81e" path="/var/lib/kubelet/pods/33a5f4d9-fe3f-40ad-904d-db5ac720c81e/volumes"
Oct 30 19:37:20.750958 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:20.750913    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-60577a-fg75m" event={"ID":"88bba8ee-11c4-48f7-80c8-d3a636ea8a1c","Type":"ContainerStarted","Data":"23ea8401fac10f668b955b6ff0e1aa427b226e306656f5385bb0220a06ec44bb"}
Oct 30 19:37:20.753922 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:20.753894    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5" event={"ID":"aa769f06-2fc2-40ce-bb1a-a198b2274afb","Type":"ContainerStarted","Data":"3766a7f3ae3915d6bb7b92255e03f826433e7da7db7ec6e8be797782bd995437"}
Oct 30 19:37:21.088072 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:21.088026    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5"
Oct 30 19:37:21.091887 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:21.091078    5043 patch_prober.go:29] interesting pod/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 30 19:37:21.091887 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:21.091152    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5" podUID="aa769f06-2fc2-40ce-bb1a-a198b2274afb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 30 19:37:21.809039 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:21.809003    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-60577a-fg75m" event={"ID":"88bba8ee-11c4-48f7-80c8-d3a636ea8a1c","Type":"ContainerStarted","Data":"a7e04db9d36e78baa44b94fe553ddd8a5cc45890bd175369a55696bc33c0e5e4"}
Oct 30 19:37:22.097277 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:22.096514    5043 patch_prober.go:29] interesting pod/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 30 19:37:22.097277 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:22.096572    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5" podUID="aa769f06-2fc2-40ce-bb1a-a198b2274afb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 30 19:37:22.820351 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:22.820309    5043 generic.go:356] "Generic (PLEG): container finished" podID="88bba8ee-11c4-48f7-80c8-d3a636ea8a1c" containerID="a7e04db9d36e78baa44b94fe553ddd8a5cc45890bd175369a55696bc33c0e5e4" exitCode=0
Oct 30 19:37:22.820850 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:22.820350    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-60577a-fg75m" event={"ID":"88bba8ee-11c4-48f7-80c8-d3a636ea8a1c","Type":"ContainerDied","Data":"a7e04db9d36e78baa44b94fe553ddd8a5cc45890bd175369a55696bc33c0e5e4"}
Oct 30 19:37:23.096376 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:23.096254    5043 patch_prober.go:29] interesting pod/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 30 19:37:23.096376 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:23.096315    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5" podUID="aa769f06-2fc2-40ce-bb1a-a198b2274afb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 30 19:37:23.099670 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:23.099221    5043 scope.go:122] "RemoveContainer" containerID="cd805e4c2201c08dadc1a5ca6678f7f3ada40453b59eeac2a454cf3d455a00cb"
Oct 30 19:37:23.105960 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:23.105937    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-60577a-fg75m"
Oct 30 19:37:23.114385 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:23.114362    5043 scope.go:122] "RemoveContainer" containerID="5463c5cdba3be37cb3047c74cdbc0542165ba1115d9855037f18559ef209a32f"
Oct 30 19:37:23.128693 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:23.128608    5043 scope.go:122] "RemoveContainer" containerID="1705f002e024c36664dde4d28c05ace7afcb0309f2b82169f0a412b0dab2869b"
Oct 30 19:37:23.143619 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:23.143506    5043 scope.go:122] "RemoveContainer" containerID="5484c8aac999fb517b0be2cdf70e7c49447abe856071f8b9eefadda51b3c7719"
Oct 30 19:37:23.160322 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:23.159874    5043 scope.go:122] "RemoveContainer" containerID="90580a5dacd0857d36bb4f7271e5e3a06bde42cb33d5e2a47c2f5fe0093155b1"
Oct 30 19:37:23.204713 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:23.204585    5043 scope.go:122] "RemoveContainer" containerID="40be87fb3f49609b445d98f54a95d3a782c0246dabf4bd81555f347cf7c08cbd"
Oct 30 19:37:23.216700 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:23.216673    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88bba8ee-11c4-48f7-80c8-d3a636ea8a1c-kube-api-access-d4mgj\" (UniqueName: \"kubernetes.io/projected/88bba8ee-11c4-48f7-80c8-d3a636ea8a1c-kube-api-access-d4mgj\") pod \"88bba8ee-11c4-48f7-80c8-d3a636ea8a1c\" (UID: \"88bba8ee-11c4-48f7-80c8-d3a636ea8a1c\") "
Oct 30 19:37:23.216843 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:23.216795    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88bba8ee-11c4-48f7-80c8-d3a636ea8a1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88bba8ee-11c4-48f7-80c8-d3a636ea8a1c-maas-api-openshift-ca\") pod \"88bba8ee-11c4-48f7-80c8-d3a636ea8a1c\" (UID: \"88bba8ee-11c4-48f7-80c8-d3a636ea8a1c\") "
Oct 30 19:37:23.216895 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:23.216870    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88bba8ee-11c4-48f7-80c8-d3a636ea8a1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88bba8ee-11c4-48f7-80c8-d3a636ea8a1c-maas-api-xks-ca\") pod \"88bba8ee-11c4-48f7-80c8-d3a636ea8a1c\" (UID: \"88bba8ee-11c4-48f7-80c8-d3a636ea8a1c\") "
Oct 30 19:37:23.217178 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:23.217152    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88bba8ee-11c4-48f7-80c8-d3a636ea8a1c-maas-api-openshift-ca" pod "88bba8ee-11c4-48f7-80c8-d3a636ea8a1c" (UID: "88bba8ee-11c4-48f7-80c8-d3a636ea8a1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:37:23.217822 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:23.217803    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88bba8ee-11c4-48f7-80c8-d3a636ea8a1c-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:23.219313 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:23.219292    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88bba8ee-11c4-48f7-80c8-d3a636ea8a1c-kube-api-access-d4mgj" pod "88bba8ee-11c4-48f7-80c8-d3a636ea8a1c" (UID: "88bba8ee-11c4-48f7-80c8-d3a636ea8a1c"). InnerVolumeSpecName "kube-api-access-d4mgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:23.219661 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:23.219638    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88bba8ee-11c4-48f7-80c8-d3a636ea8a1c-maas-api-xks-ca" pod "88bba8ee-11c4-48f7-80c8-d3a636ea8a1c" (UID: "88bba8ee-11c4-48f7-80c8-d3a636ea8a1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:37:23.220496 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:23.220477    5043 scope.go:122] "RemoveContainer" containerID="d4c089655e949581d4ef453c301ac4802f81fea906ed0fdc8b15d56d4def256f"
Oct 30 19:37:23.300356 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:23.300054    5043 scope.go:122] "RemoveContainer" containerID="3d7cebf2c5881af085ccfbdfd8b1b13d028224eb99c9b18abe4b12b498c83d4d"
Oct 30 19:37:23.321050 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:23.319342    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4mgj\" (UniqueName: \"kubernetes.io/projected/88bba8ee-11c4-48f7-80c8-d3a636ea8a1c-kube-api-access-d4mgj\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:23.321050 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:23.319377    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88bba8ee-11c4-48f7-80c8-d3a636ea8a1c-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:23.325913 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:23.325887    5043 scope.go:122] "RemoveContainer" containerID="0ecf41f512dd96fa1fba76b270d68443bc3faed4e59292f2daa9301bae6bd671"
Oct 30 19:37:23.354712 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:23.354689    5043 scope.go:122] "RemoveContainer" containerID="30de05097ec1b9339ebde00cd70dbf95e988c97f28557320bbdf2482d1e87ac5"
Oct 30 19:37:23.492703 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:23.492567    5043 scope.go:122] "RemoveContainer" containerID="573b34198874a4f72afaa00c9477052d3ac1c32dd1522437c3dba33b5b09a608"
Oct 30 19:37:23.831964 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:23.831939    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-60577a-fg75m"
Oct 30 19:37:23.832350 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:23.831978    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-60577a-fg75m" event={"ID":"88bba8ee-11c4-48f7-80c8-d3a636ea8a1c","Type":"ContainerDied","Data":"23ea8401fac10f668b955b6ff0e1aa427b226e306656f5385bb0220a06ec44bb"}
Oct 30 19:37:23.832350 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:23.832016    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23ea8401fac10f668b955b6ff0e1aa427b226e306656f5385bb0220a06ec44bb"
Oct 30 19:37:24.084154 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:24.084113    5043 patch_prober.go:29] interesting pod/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 30 19:37:24.084477 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:24.084446    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5" podUID="aa769f06-2fc2-40ce-bb1a-a198b2274afb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 30 19:37:24.404747 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:24.404644    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-73f96d-65c48fbd9f-98p49" podUID="6f7e3a16-0761-42c5-9797-85fe755780d8" containerName="payload-pre-processing" containerID="cri-o://5903d9e55ac21661da7bcd1afcbfc204b75f016071db86a6cfd09ac53e499a01" gracePeriod=25
Oct 30 19:37:24.710790 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:24.710373    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5"
Oct 30 19:37:24.806728 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:24.806701    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-73f96d-65c48fbd9f-98p49"
Oct 30 19:37:24.845410 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:24.845365    5043 generic.go:356] "Generic (PLEG): container finished" podID="6f7e3a16-0761-42c5-9797-85fe755780d8" containerID="5903d9e55ac21661da7bcd1afcbfc204b75f016071db86a6cfd09ac53e499a01" exitCode=0
Oct 30 19:37:24.845881 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:24.845418    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-73f96d-65c48fbd9f-98p49"
Oct 30 19:37:24.845881 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:24.845448    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-73f96d-65c48fbd9f-98p49" event={"ID":"6f7e3a16-0761-42c5-9797-85fe755780d8","Type":"ContainerDied","Data":"5903d9e55ac21661da7bcd1afcbfc204b75f016071db86a6cfd09ac53e499a01"}
Oct 30 19:37:24.845881 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:24.845504    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-73f96d-65c48fbd9f-98p49" event={"ID":"6f7e3a16-0761-42c5-9797-85fe755780d8","Type":"ContainerDied","Data":"8f023eaff116fc0bfe86293ef4c71ab9897c86ca30273ee7eea1486925565fc2"}
Oct 30 19:37:24.845881 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:24.845532    5043 scope.go:122] "RemoveContainer" containerID="5903d9e55ac21661da7bcd1afcbfc204b75f016071db86a6cfd09ac53e499a01"
Oct 30 19:37:24.889942 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:24.887531    5043 scope.go:122] "RemoveContainer" containerID="5903d9e55ac21661da7bcd1afcbfc204b75f016071db86a6cfd09ac53e499a01"
Oct 30 19:37:24.889942 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:37:24.888024    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5903d9e55ac21661da7bcd1afcbfc204b75f016071db86a6cfd09ac53e499a01\": container with ID starting with 5903d9e55ac21661da7bcd1afcbfc204b75f016071db86a6cfd09ac53e499a01 not found: ID does not exist" containerID="5903d9e55ac21661da7bcd1afcbfc204b75f016071db86a6cfd09ac53e499a01"
Oct 30 19:37:24.889942 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:24.888058    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5903d9e55ac21661da7bcd1afcbfc204b75f016071db86a6cfd09ac53e499a01"} err="failed to get container status \"5903d9e55ac21661da7bcd1afcbfc204b75f016071db86a6cfd09ac53e499a01\": rpc error: code = NotFound desc = could not find container \"5903d9e55ac21661da7bcd1afcbfc204b75f016071db86a6cfd09ac53e499a01\": container with ID starting with 5903d9e55ac21661da7bcd1afcbfc204b75f016071db86a6cfd09ac53e499a01 not found: ID does not exist"
Oct 30 19:37:24.954010 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:24.953977    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f7e3a16-0761-42c5-9797-85fe755780d8-kube-api-access-nn572\" (UniqueName: \"kubernetes.io/projected/6f7e3a16-0761-42c5-9797-85fe755780d8-kube-api-access-nn572\") pod \"6f7e3a16-0761-42c5-9797-85fe755780d8\" (UID: \"6f7e3a16-0761-42c5-9797-85fe755780d8\") "
Oct 30 19:37:24.954278 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:24.954259    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f7e3a16-0761-42c5-9797-85fe755780d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f7e3a16-0761-42c5-9797-85fe755780d8-plugins-config-volume\") pod \"6f7e3a16-0761-42c5-9797-85fe755780d8\" (UID: \"6f7e3a16-0761-42c5-9797-85fe755780d8\") "
Oct 30 19:37:24.954673 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:24.954594    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f7e3a16-0761-42c5-9797-85fe755780d8-plugins-config-volume" pod "6f7e3a16-0761-42c5-9797-85fe755780d8" (UID: "6f7e3a16-0761-42c5-9797-85fe755780d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:37:24.955062 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:24.955042    5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f7e3a16-0761-42c5-9797-85fe755780d8-plugins-config-volume\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:24.988997 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:24.988958    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f7e3a16-0761-42c5-9797-85fe755780d8-kube-api-access-nn572" pod "6f7e3a16-0761-42c5-9797-85fe755780d8" (UID: "6f7e3a16-0761-42c5-9797-85fe755780d8"). InnerVolumeSpecName "kube-api-access-nn572". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:25.015066 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:37:25.015026    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:37:25.078761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:25.057721    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn572\" (UniqueName: \"kubernetes.io/projected/6f7e3a16-0761-42c5-9797-85fe755780d8-kube-api-access-nn572\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:25.083351 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:25.083305    5043 patch_prober.go:29] interesting pod/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 30 19:37:25.083517 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:25.083368    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5" podUID="aa769f06-2fc2-40ce-bb1a-a198b2274afb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 30 19:37:25.244526 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:25.244411    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-73f96d-65c48fbd9f-98p49"]
Oct 30 19:37:25.250376 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:25.250346    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-73f96d-65c48fbd9f-98p49"]
Oct 30 19:37:26.102765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:26.088258    5043 patch_prober.go:29] interesting pod/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 30 19:37:26.102765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:26.089102    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5" podUID="aa769f06-2fc2-40ce-bb1a-a198b2274afb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 30 19:37:26.426061 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:26.425916    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-95d2ac-ctr48"]
Oct 30 19:37:26.433710 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:26.433678    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-95d2ac-ctr48"]
Oct 30 19:37:26.666496 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:26.665804    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34027f03-ad24-4f28-b53d-8e21422dac2e" path="/var/lib/kubelet/pods/34027f03-ad24-4f28-b53d-8e21422dac2e/volumes"
Oct 30 19:37:26.666713 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:26.666672    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f7e3a16-0761-42c5-9797-85fe755780d8" path="/var/lib/kubelet/pods/6f7e3a16-0761-42c5-9797-85fe755780d8/volumes"
Oct 30 19:37:27.086479 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:27.086446    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5"
Oct 30 19:37:27.087939 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:27.087916    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5"
Oct 30 19:37:27.558560 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:27.558523    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9"]
Oct 30 19:37:27.559126 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:27.558777    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9" podUID="224648b8-553f-44ad-9d9a-3784a7bea898" containerName="istio-proxy" containerID="cri-o://4ea546a3bab408e0b58ecbeeab9f86306a7f29763425dcad8cdf95930b5cb1a4" gracePeriod=30
Oct 30 19:37:27.564408 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:27.564369    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg"]
Oct 30 19:37:27.564821 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:27.564787    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg" podUID="a851569d-8aa4-4af2-9f3c-c90c4895b95a" containerName="istio-proxy" containerID="cri-o://a6b31cdbbfe4e45387fe8fd60975db8818bece18976ce5c683d46c3fd9affd72" gracePeriod=30
Oct 30 19:37:27.574759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:27.572106    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q"]
Oct 30 19:37:27.574759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:27.572332    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q" podUID="d8a7200d-6acf-4008-8156-d9102730ac76" containerName="istio-proxy" containerID="cri-o://200b1ca9a3dfa73fa6d763f3b5eefb5d256b363ec4d4ae282a751afd3c04377d" gracePeriod=30
Oct 30 19:37:27.579989 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:27.579959    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf"]
Oct 30 19:37:27.580203 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:27.580176    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf" podUID="4fea9e2a-77c0-48c4-8517-3e104fc043b8" containerName="istio-proxy" containerID="cri-o://a4d973530c2d7a358d00dfa3f48bfbea57b1ba2a3b9dd90f577c1b97cfccf368" gracePeriod=30
Oct 30 19:37:27.642125 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:27.642091    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-82012b-5987f89c59-xqtgs"
Oct 30 19:37:27.740202 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:27.740164    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j"]
Oct 30 19:37:27.740399 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:27.740377    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j" podUID="c3185a2a-d994-41d4-a8d7-9b7e966b9f1c" containerName="istio-proxy" containerID="cri-o://20e7692b745948d12f8044909a2acb7baae7326b2556c005133d80f20e87bf91" gracePeriod=30
Oct 30 19:37:27.919707 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:27.919627    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt"]
Oct 30 19:37:27.919895 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:27.919864    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt" podUID="359c47fc-f8af-43f8-865b-1f187fb2672c" containerName="istio-proxy" containerID="cri-o://2b329b14627cf06b423a2cbca22183d9d506012c6472357ef35919dc3b4171f4" gracePeriod=30
Oct 30 19:37:28.068389 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:28.068355    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7"]
Oct 30 19:37:28.068579 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:28.068539    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7" podUID="8ce15b3c-1548-489c-a4f1-2fc151a53d35" containerName="istio-proxy" containerID="cri-o://5680ee690c20c4258226d6037905090e6336e4a1a7f3ec1f7c9c35d6924fd60c" gracePeriod=30
Oct 30 19:37:29.671878 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:29.671845    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89a3f6-db46ffd87-8l4s5"
Oct 30 19:37:31.110824 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:31.110792    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk"]
Oct 30 19:37:31.111703 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:31.111683    5043 cpu_manager.go:414] "containerMap: removing container" podUID="6f7e3a16-0761-42c5-9797-85fe755780d8" containerName="payload-pre-processing"
Oct 30 19:37:31.111703 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:31.111702    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f7e3a16-0761-42c5-9797-85fe755780d8" containerName="payload-pre-processing"
Oct 30 19:37:31.111886 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:31.111719    5043 cpu_manager.go:414] "containerMap: removing container" podUID="88bba8ee-11c4-48f7-80c8-d3a636ea8a1c" containerName="revoke-keys"
Oct 30 19:37:31.111886 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:31.111725    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88bba8ee-11c4-48f7-80c8-d3a636ea8a1c" containerName="revoke-keys"
Oct 30 19:37:31.111886 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:31.111867    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f7e3a16-0761-42c5-9797-85fe755780d8" containerName="payload-pre-processing"
Oct 30 19:37:31.111886 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:31.111879    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="88bba8ee-11c4-48f7-80c8-d3a636ea8a1c" containerName="revoke-keys"
Oct 30 19:37:31.117694 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:31.117671    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk"
Oct 30 19:37:31.144963 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:31.131869    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk"]
Oct 30 19:37:31.210985 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:31.210951    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ae382e3-be21-4e69-bf70-a8c0b8517789-istio-envoy\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk\" (UID: \"6ae382e3-be21-4e69-bf70-a8c0b8517789\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk"
Oct 30 19:37:31.211160 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:31.211132    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ae382e3-be21-4e69-bf70-a8c0b8517789-istiod-ca-cert\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk\" (UID: \"6ae382e3-be21-4e69-bf70-a8c0b8517789\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk"
Oct 30 19:37:31.211338 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:31.211304    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae382e3-be21-4e69-bf70-a8c0b8517789-credential-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk\" (UID: \"6ae382e3-be21-4e69-bf70-a8c0b8517789\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk"
Oct 30 19:37:31.211409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:31.211363    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ae382e3-be21-4e69-bf70-a8c0b8517789-istio-data\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk\" (UID: \"6ae382e3-be21-4e69-bf70-a8c0b8517789\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk"
Oct 30 19:37:31.211513 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:31.211496    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae382e3-be21-4e69-bf70-a8c0b8517789-workload-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk\" (UID: \"6ae382e3-be21-4e69-bf70-a8c0b8517789\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk"
Oct 30 19:37:31.211548 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:31.211525    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ae382e3-be21-4e69-bf70-a8c0b8517789-istio-token\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk\" (UID: \"6ae382e3-be21-4e69-bf70-a8c0b8517789\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk"
Oct 30 19:37:31.211585 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:31.211575    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ae382e3-be21-4e69-bf70-a8c0b8517789-istio-podinfo\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk\" (UID: \"6ae382e3-be21-4e69-bf70-a8c0b8517789\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk"
Oct 30 19:37:31.211679 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:31.211665    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmbk6\" (UniqueName: \"kubernetes.io/projected/6ae382e3-be21-4e69-bf70-a8c0b8517789-kube-api-access-xmbk6\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk\" (UID: \"6ae382e3-be21-4e69-bf70-a8c0b8517789\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk"
Oct 30 19:37:31.211804 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:31.211783    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ae382e3-be21-4e69-bf70-a8c0b8517789-workload-certs\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk\" (UID: \"6ae382e3-be21-4e69-bf70-a8c0b8517789\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk"
Oct 30 19:37:31.313370 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:31.313329    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae382e3-be21-4e69-bf70-a8c0b8517789-credential-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk\" (UID: \"6ae382e3-be21-4e69-bf70-a8c0b8517789\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk"
Oct 30 19:37:31.313370 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:31.313378    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ae382e3-be21-4e69-bf70-a8c0b8517789-istio-data\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk\" (UID: \"6ae382e3-be21-4e69-bf70-a8c0b8517789\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk"
Oct 30 19:37:31.313648 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:31.313475    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae382e3-be21-4e69-bf70-a8c0b8517789-workload-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk\" (UID: \"6ae382e3-be21-4e69-bf70-a8c0b8517789\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk"
Oct 30 19:37:31.313648 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:31.313497    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ae382e3-be21-4e69-bf70-a8c0b8517789-istio-token\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk\" (UID: \"6ae382e3-be21-4e69-bf70-a8c0b8517789\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk"
Oct 30 19:37:31.313648 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:31.313532    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ae382e3-be21-4e69-bf70-a8c0b8517789-istio-podinfo\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk\" (UID: \"6ae382e3-be21-4e69-bf70-a8c0b8517789\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk"
Oct 30 19:37:31.313648 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:31.313586    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmbk6\" (UniqueName: \"kubernetes.io/projected/6ae382e3-be21-4e69-bf70-a8c0b8517789-kube-api-access-xmbk6\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk\" (UID: \"6ae382e3-be21-4e69-bf70-a8c0b8517789\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk"
Oct 30 19:37:31.313839 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:31.313658    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ae382e3-be21-4e69-bf70-a8c0b8517789-workload-certs\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk\" (UID: \"6ae382e3-be21-4e69-bf70-a8c0b8517789\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk"
Oct 30 19:37:31.313839 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:31.313755    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ae382e3-be21-4e69-bf70-a8c0b8517789-istio-envoy\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk\" (UID: \"6ae382e3-be21-4e69-bf70-a8c0b8517789\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk"
Oct 30 19:37:31.313839 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:31.313824    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ae382e3-be21-4e69-bf70-a8c0b8517789-istiod-ca-cert\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk\" (UID: \"6ae382e3-be21-4e69-bf70-a8c0b8517789\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk"
Oct 30 19:37:31.313960 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:31.313835    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae382e3-be21-4e69-bf70-a8c0b8517789-credential-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk\" (UID: \"6ae382e3-be21-4e69-bf70-a8c0b8517789\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk"
Oct 30 19:37:31.313960 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:31.313871    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae382e3-be21-4e69-bf70-a8c0b8517789-workload-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk\" (UID: \"6ae382e3-be21-4e69-bf70-a8c0b8517789\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk"
Oct 30 19:37:31.313960 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:31.313925    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ae382e3-be21-4e69-bf70-a8c0b8517789-istio-data\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk\" (UID: \"6ae382e3-be21-4e69-bf70-a8c0b8517789\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk"
Oct 30 19:37:31.314279 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:31.314249    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ae382e3-be21-4e69-bf70-a8c0b8517789-workload-certs\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk\" (UID: \"6ae382e3-be21-4e69-bf70-a8c0b8517789\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk"
Oct 30 19:37:31.314466 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:31.314447    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ae382e3-be21-4e69-bf70-a8c0b8517789-istiod-ca-cert\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk\" (UID: \"6ae382e3-be21-4e69-bf70-a8c0b8517789\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk"
Oct 30 19:37:31.316454 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:31.316430    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ae382e3-be21-4e69-bf70-a8c0b8517789-istio-envoy\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk\" (UID: \"6ae382e3-be21-4e69-bf70-a8c0b8517789\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk"
Oct 30 19:37:31.316880 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:31.316847    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ae382e3-be21-4e69-bf70-a8c0b8517789-istio-podinfo\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk\" (UID: \"6ae382e3-be21-4e69-bf70-a8c0b8517789\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk"
Oct 30 19:37:31.321970 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:31.321948    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ae382e3-be21-4e69-bf70-a8c0b8517789-istio-token\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk\" (UID: \"6ae382e3-be21-4e69-bf70-a8c0b8517789\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk"
Oct 30 19:37:31.322301 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:31.322273    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmbk6\" (UniqueName: \"kubernetes.io/projected/6ae382e3-be21-4e69-bf70-a8c0b8517789-kube-api-access-xmbk6\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk\" (UID: \"6ae382e3-be21-4e69-bf70-a8c0b8517789\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk"
Oct 30 19:37:31.353606 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:31.353566    5043 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:37:31.353820 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:31.353645    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9" podUID="224648b8-553f-44ad-9d9a-3784a7bea898" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:37:31.441220 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:31.441139    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk"
Oct 30 19:37:31.887353 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:31.880522    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk"]
Oct 30 19:37:31.887786 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:31.887717    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:37:31.887873 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:31.887830    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:37:31.887937 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:31.887875    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:37:31.965969 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:31.965938    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk" event={"ID":"6ae382e3-be21-4e69-bf70-a8c0b8517789","Type":"ContainerStarted","Data":"7ef9ee0386e42e0994294082c8014ff993bb3e400357e96170b803fa4eff9a35"}
Oct 30 19:37:33.087092 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:33.076377    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-194e54e4-9dd4p"]
Oct 30 19:37:33.139014 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:33.138932    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-194e54e4-9dd4p"]
Oct 30 19:37:33.139187 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:33.139057    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-194e54e4-9dd4p"
Oct 30 19:37:33.147059 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:33.147025    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:37:33.147759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:33.147719    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:37:33.147878 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:33.147813    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:37:33.147933 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:33.147911    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:37:33.148661 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:33.148638    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6266b58-dd5hf"]
Oct 30 19:37:33.162764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:33.155724    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk" event={"ID":"6ae382e3-be21-4e69-bf70-a8c0b8517789","Type":"ContainerStarted","Data":"d0f492c58f63fcfc184b1ecf6d0a1fe3c10c76f34de4f2a24f54e62573e9b564"}
Oct 30 19:37:33.162764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:33.161870    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6266b58-dd5hf"
Oct 30 19:37:33.207817 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:33.178714    5043 generic.go:356] "Generic (PLEG): container finished" podID="224648b8-553f-44ad-9d9a-3784a7bea898" containerID="4ea546a3bab408e0b58ecbeeab9f86306a7f29763425dcad8cdf95930b5cb1a4" exitCode=0
Oct 30 19:37:33.207817 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:33.179009    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9" event={"ID":"224648b8-553f-44ad-9d9a-3784a7bea898","Type":"ContainerDied","Data":"4ea546a3bab408e0b58ecbeeab9f86306a7f29763425dcad8cdf95930b5cb1a4"}
Oct 30 19:37:33.207817 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:33.179069    5043 scope.go:122] "RemoveContainer" containerID="8504037de69583964b9cded037a0e8cca9e67eb62d1504d8e2ad1745a0effd54"
Oct 30 19:37:33.219495 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:33.219463    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6266b58-dd5hf"]
Oct 30 19:37:33.233098 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:33.233065    5043 generic.go:356] "Generic (PLEG): container finished" podID="c3185a2a-d994-41d4-a8d7-9b7e966b9f1c" containerID="20e7692b745948d12f8044909a2acb7baae7326b2556c005133d80f20e87bf91" exitCode=0
Oct 30 19:37:33.233329 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:33.233172    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j" event={"ID":"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c","Type":"ContainerDied","Data":"20e7692b745948d12f8044909a2acb7baae7326b2556c005133d80f20e87bf91"}
Oct 30 19:37:33.235969 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:33.235825    5043 generic.go:356] "Generic (PLEG): container finished" podID="4fea9e2a-77c0-48c4-8517-3e104fc043b8" containerID="a4d973530c2d7a358d00dfa3f48bfbea57b1ba2a3b9dd90f577c1b97cfccf368" exitCode=0
Oct 30 19:37:33.235969 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:33.235911    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf" event={"ID":"4fea9e2a-77c0-48c4-8517-3e104fc043b8","Type":"ContainerDied","Data":"a4d973530c2d7a358d00dfa3f48bfbea57b1ba2a3b9dd90f577c1b97cfccf368"}
Oct 30 19:37:33.239514 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:33.239368    5043 generic.go:356] "Generic (PLEG): container finished" podID="d8a7200d-6acf-4008-8156-d9102730ac76" containerID="200b1ca9a3dfa73fa6d763f3b5eefb5d256b363ec4d4ae282a751afd3c04377d" exitCode=0
Oct 30 19:37:33.239514 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:33.239463    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q" event={"ID":"d8a7200d-6acf-4008-8156-d9102730ac76","Type":"ContainerDied","Data":"200b1ca9a3dfa73fa6d763f3b5eefb5d256b363ec4d4ae282a751afd3c04377d"}
Oct 30 19:37:33.265759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:33.261927    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87ce476f-991e-4437-a15e-f7410d2392a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6266b58-dd5hf\" (UID: \"87ce476f-991e-4437-a15e-f7410d2392a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6266b58-dd5hf"
Oct 30 19:37:33.265759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:33.261979    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08851e5a-2c32-446b-9ffa-7508ac46cf2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-194e54e4-9dd4p\" (UID: \"08851e5a-2c32-446b-9ffa-7508ac46cf2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-194e54e4-9dd4p"
Oct 30 19:37:33.265759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:33.262249    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jcl7\" (UniqueName: \"kubernetes.io/projected/08851e5a-2c32-446b-9ffa-7508ac46cf2c-kube-api-access-5jcl7\") pod \"maas-api-revoke-sub-194e54e4-9dd4p\" (UID: \"08851e5a-2c32-446b-9ffa-7508ac46cf2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-194e54e4-9dd4p"
Oct 30 19:37:33.265759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:33.262288    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08851e5a-2c32-446b-9ffa-7508ac46cf2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-194e54e4-9dd4p\" (UID: \"08851e5a-2c32-446b-9ffa-7508ac46cf2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-194e54e4-9dd4p"
Oct 30 19:37:33.265759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:33.262408    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87ce476f-991e-4437-a15e-f7410d2392a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6266b58-dd5hf\" (UID: \"87ce476f-991e-4437-a15e-f7410d2392a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6266b58-dd5hf"
Oct 30 19:37:33.265759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:33.262566    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8rjn\" (UniqueName: \"kubernetes.io/projected/87ce476f-991e-4437-a15e-f7410d2392a5-kube-api-access-m8rjn\") pod \"maas-api-revoke-sub-e6266b58-dd5hf\" (UID: \"87ce476f-991e-4437-a15e-f7410d2392a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6266b58-dd5hf"
Oct 30 19:37:33.388899 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:33.386516    5043 generic.go:356] "Generic (PLEG): container finished" podID="a851569d-8aa4-4af2-9f3c-c90c4895b95a" containerID="a6b31cdbbfe4e45387fe8fd60975db8818bece18976ce5c683d46c3fd9affd72" exitCode=0
Oct 30 19:37:33.388899 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:33.386645    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg" event={"ID":"a851569d-8aa4-4af2-9f3c-c90c4895b95a","Type":"ContainerDied","Data":"a6b31cdbbfe4e45387fe8fd60975db8818bece18976ce5c683d46c3fd9affd72"}
Oct 30 19:37:33.391862 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:33.389782    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8rjn\" (UniqueName: \"kubernetes.io/projected/87ce476f-991e-4437-a15e-f7410d2392a5-kube-api-access-m8rjn\") pod \"maas-api-revoke-sub-e6266b58-dd5hf\" (UID: \"87ce476f-991e-4437-a15e-f7410d2392a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6266b58-dd5hf"
Oct 30 19:37:33.391862 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:33.389864    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87ce476f-991e-4437-a15e-f7410d2392a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6266b58-dd5hf\" (UID: \"87ce476f-991e-4437-a15e-f7410d2392a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6266b58-dd5hf"
Oct 30 19:37:33.391862 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:33.389901    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08851e5a-2c32-446b-9ffa-7508ac46cf2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-194e54e4-9dd4p\" (UID: \"08851e5a-2c32-446b-9ffa-7508ac46cf2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-194e54e4-9dd4p"
Oct 30 19:37:33.391862 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:33.390147    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jcl7\" (UniqueName: \"kubernetes.io/projected/08851e5a-2c32-446b-9ffa-7508ac46cf2c-kube-api-access-5jcl7\") pod \"maas-api-revoke-sub-194e54e4-9dd4p\" (UID: \"08851e5a-2c32-446b-9ffa-7508ac46cf2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-194e54e4-9dd4p"
Oct 30 19:37:33.391862 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:33.390184    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08851e5a-2c32-446b-9ffa-7508ac46cf2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-194e54e4-9dd4p\" (UID: \"08851e5a-2c32-446b-9ffa-7508ac46cf2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-194e54e4-9dd4p"
Oct 30 19:37:33.391862 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:33.390298    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87ce476f-991e-4437-a15e-f7410d2392a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6266b58-dd5hf\" (UID: \"87ce476f-991e-4437-a15e-f7410d2392a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6266b58-dd5hf"
Oct 30 19:37:33.391862 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:33.391121    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87ce476f-991e-4437-a15e-f7410d2392a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6266b58-dd5hf\" (UID: \"87ce476f-991e-4437-a15e-f7410d2392a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6266b58-dd5hf"
Oct 30 19:37:33.393761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:33.393348    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08851e5a-2c32-446b-9ffa-7508ac46cf2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-194e54e4-9dd4p\" (UID: \"08851e5a-2c32-446b-9ffa-7508ac46cf2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-194e54e4-9dd4p"
Oct 30 19:37:33.419803 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:33.417168    5043 generic.go:356] "Generic (PLEG): container finished" podID="359c47fc-f8af-43f8-865b-1f187fb2672c" containerID="2b329b14627cf06b423a2cbca22183d9d506012c6472357ef35919dc3b4171f4" exitCode=0
Oct 30 19:37:33.419803 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:33.417224    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt" event={"ID":"359c47fc-f8af-43f8-865b-1f187fb2672c","Type":"ContainerDied","Data":"2b329b14627cf06b423a2cbca22183d9d506012c6472357ef35919dc3b4171f4"}
Oct 30 19:37:33.425445 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:33.425412    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jcl7\" (UniqueName: \"kubernetes.io/projected/08851e5a-2c32-446b-9ffa-7508ac46cf2c-kube-api-access-5jcl7\") pod \"maas-api-revoke-sub-194e54e4-9dd4p\" (UID: \"08851e5a-2c32-446b-9ffa-7508ac46cf2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-194e54e4-9dd4p"
Oct 30 19:37:33.436160 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:33.436124    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08851e5a-2c32-446b-9ffa-7508ac46cf2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-194e54e4-9dd4p\" (UID: \"08851e5a-2c32-446b-9ffa-7508ac46cf2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-194e54e4-9dd4p"
Oct 30 19:37:33.436860 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:33.436833    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87ce476f-991e-4437-a15e-f7410d2392a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6266b58-dd5hf\" (UID: \"87ce476f-991e-4437-a15e-f7410d2392a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6266b58-dd5hf"
Oct 30 19:37:33.443515 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:33.443476    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk"
Oct 30 19:37:33.443714 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:33.443595    5043 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Oct 30 19:37:33.443714 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:33.443637    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk" podUID="6ae382e3-be21-4e69-bf70-a8c0b8517789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Oct 30 19:37:33.446527 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:33.446435    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8rjn\" (UniqueName: \"kubernetes.io/projected/87ce476f-991e-4437-a15e-f7410d2392a5-kube-api-access-m8rjn\") pod \"maas-api-revoke-sub-e6266b58-dd5hf\" (UID: \"87ce476f-991e-4437-a15e-f7410d2392a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6266b58-dd5hf"
Oct 30 19:37:33.654163 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:33.638988    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6266b58-dd5hf"
Oct 30 19:37:33.657767 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:33.657328    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-194e54e4-9dd4p"
Oct 30 19:37:34.076764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.062210    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf"
Oct 30 19:37:34.107758 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.104761    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt"
Oct 30 19:37:34.111515 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.111483    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fea9e2a-77c0-48c4-8517-3e104fc043b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fea9e2a-77c0-48c4-8517-3e104fc043b8-credential-socket\") pod \"4fea9e2a-77c0-48c4-8517-3e104fc043b8\" (UID: \"4fea9e2a-77c0-48c4-8517-3e104fc043b8\") "
Oct 30 19:37:34.111636 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.111541    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/359c47fc-f8af-43f8-865b-1f187fb2672c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/359c47fc-f8af-43f8-865b-1f187fb2672c-workload-socket\") pod \"359c47fc-f8af-43f8-865b-1f187fb2672c\" (UID: \"359c47fc-f8af-43f8-865b-1f187fb2672c\") "
Oct 30 19:37:34.111636 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.111588    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fea9e2a-77c0-48c4-8517-3e104fc043b8-istio-token\" (UniqueName: \"kubernetes.io/projected/4fea9e2a-77c0-48c4-8517-3e104fc043b8-istio-token\") pod \"4fea9e2a-77c0-48c4-8517-3e104fc043b8\" (UID: \"4fea9e2a-77c0-48c4-8517-3e104fc043b8\") "
Oct 30 19:37:34.111762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.111642    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4fea9e2a-77c0-48c4-8517-3e104fc043b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fea9e2a-77c0-48c4-8517-3e104fc043b8-istio-podinfo\") pod \"4fea9e2a-77c0-48c4-8517-3e104fc043b8\" (UID: \"4fea9e2a-77c0-48c4-8517-3e104fc043b8\") "
Oct 30 19:37:34.111762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.111669    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fea9e2a-77c0-48c4-8517-3e104fc043b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fea9e2a-77c0-48c4-8517-3e104fc043b8-workload-socket\") pod \"4fea9e2a-77c0-48c4-8517-3e104fc043b8\" (UID: \"4fea9e2a-77c0-48c4-8517-3e104fc043b8\") "
Oct 30 19:37:34.111880 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.111802    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fea9e2a-77c0-48c4-8517-3e104fc043b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fea9e2a-77c0-48c4-8517-3e104fc043b8-istio-data\") pod \"4fea9e2a-77c0-48c4-8517-3e104fc043b8\" (UID: \"4fea9e2a-77c0-48c4-8517-3e104fc043b8\") "
Oct 30 19:37:34.111880 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.111833    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/359c47fc-f8af-43f8-865b-1f187fb2672c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/359c47fc-f8af-43f8-865b-1f187fb2672c-istio-envoy\") pod \"359c47fc-f8af-43f8-865b-1f187fb2672c\" (UID: \"359c47fc-f8af-43f8-865b-1f187fb2672c\") "
Oct 30 19:37:34.111977 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.111879    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fea9e2a-77c0-48c4-8517-3e104fc043b8-kube-api-access-mzbfz\" (UniqueName: \"kubernetes.io/projected/4fea9e2a-77c0-48c4-8517-3e104fc043b8-kube-api-access-mzbfz\") pod \"4fea9e2a-77c0-48c4-8517-3e104fc043b8\" (UID: \"4fea9e2a-77c0-48c4-8517-3e104fc043b8\") "
Oct 30 19:37:34.111977 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.111924    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fea9e2a-77c0-48c4-8517-3e104fc043b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fea9e2a-77c0-48c4-8517-3e104fc043b8-istiod-ca-cert\") pod \"4fea9e2a-77c0-48c4-8517-3e104fc043b8\" (UID: \"4fea9e2a-77c0-48c4-8517-3e104fc043b8\") "
Oct 30 19:37:34.111977 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.111962    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/359c47fc-f8af-43f8-865b-1f187fb2672c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/359c47fc-f8af-43f8-865b-1f187fb2672c-istio-data\") pod \"359c47fc-f8af-43f8-865b-1f187fb2672c\" (UID: \"359c47fc-f8af-43f8-865b-1f187fb2672c\") "
Oct 30 19:37:34.112124 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.112025    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/359c47fc-f8af-43f8-865b-1f187fb2672c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/359c47fc-f8af-43f8-865b-1f187fb2672c-istiod-ca-cert\") pod \"359c47fc-f8af-43f8-865b-1f187fb2672c\" (UID: \"359c47fc-f8af-43f8-865b-1f187fb2672c\") "
Oct 30 19:37:34.112124 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.112065    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/359c47fc-f8af-43f8-865b-1f187fb2672c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/359c47fc-f8af-43f8-865b-1f187fb2672c-workload-certs\") pod \"359c47fc-f8af-43f8-865b-1f187fb2672c\" (UID: \"359c47fc-f8af-43f8-865b-1f187fb2672c\") "
Oct 30 19:37:34.112124 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.112104    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fea9e2a-77c0-48c4-8517-3e104fc043b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fea9e2a-77c0-48c4-8517-3e104fc043b8-istio-envoy\") pod \"4fea9e2a-77c0-48c4-8517-3e104fc043b8\" (UID: \"4fea9e2a-77c0-48c4-8517-3e104fc043b8\") "
Oct 30 19:37:34.112264 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.112176    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/359c47fc-f8af-43f8-865b-1f187fb2672c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/359c47fc-f8af-43f8-865b-1f187fb2672c-istio-podinfo\") pod \"359c47fc-f8af-43f8-865b-1f187fb2672c\" (UID: \"359c47fc-f8af-43f8-865b-1f187fb2672c\") "
Oct 30 19:37:34.112317 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.112274    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/359c47fc-f8af-43f8-865b-1f187fb2672c-kube-api-access-989kw\" (UniqueName: \"kubernetes.io/projected/359c47fc-f8af-43f8-865b-1f187fb2672c-kube-api-access-989kw\") pod \"359c47fc-f8af-43f8-865b-1f187fb2672c\" (UID: \"359c47fc-f8af-43f8-865b-1f187fb2672c\") "
Oct 30 19:37:34.112317 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.112304    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/359c47fc-f8af-43f8-865b-1f187fb2672c-istio-token\" (UniqueName: \"kubernetes.io/projected/359c47fc-f8af-43f8-865b-1f187fb2672c-istio-token\") pod \"359c47fc-f8af-43f8-865b-1f187fb2672c\" (UID: \"359c47fc-f8af-43f8-865b-1f187fb2672c\") "
Oct 30 19:37:34.112416 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.112375    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/359c47fc-f8af-43f8-865b-1f187fb2672c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/359c47fc-f8af-43f8-865b-1f187fb2672c-credential-socket\") pod \"359c47fc-f8af-43f8-865b-1f187fb2672c\" (UID: \"359c47fc-f8af-43f8-865b-1f187fb2672c\") "
Oct 30 19:37:34.112522 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.112505    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fea9e2a-77c0-48c4-8517-3e104fc043b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fea9e2a-77c0-48c4-8517-3e104fc043b8-workload-certs\") pod \"4fea9e2a-77c0-48c4-8517-3e104fc043b8\" (UID: \"4fea9e2a-77c0-48c4-8517-3e104fc043b8\") "
Oct 30 19:37:34.113987 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.113824    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/359c47fc-f8af-43f8-865b-1f187fb2672c-workload-socket" pod "359c47fc-f8af-43f8-865b-1f187fb2672c" (UID: "359c47fc-f8af-43f8-865b-1f187fb2672c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:34.114069 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.114027    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fea9e2a-77c0-48c4-8517-3e104fc043b8-credential-socket" pod "4fea9e2a-77c0-48c4-8517-3e104fc043b8" (UID: "4fea9e2a-77c0-48c4-8517-3e104fc043b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:34.114292 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.114253    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fea9e2a-77c0-48c4-8517-3e104fc043b8-istio-data" pod "4fea9e2a-77c0-48c4-8517-3e104fc043b8" (UID: "4fea9e2a-77c0-48c4-8517-3e104fc043b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:34.114458 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.114438    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fea9e2a-77c0-48c4-8517-3e104fc043b8-workload-socket" pod "4fea9e2a-77c0-48c4-8517-3e104fc043b8" (UID: "4fea9e2a-77c0-48c4-8517-3e104fc043b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:34.114585 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.114564    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fea9e2a-77c0-48c4-8517-3e104fc043b8-istiod-ca-cert" pod "4fea9e2a-77c0-48c4-8517-3e104fc043b8" (UID: "4fea9e2a-77c0-48c4-8517-3e104fc043b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:37:34.118557 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.118530    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fea9e2a-77c0-48c4-8517-3e104fc043b8-workload-certs" pod "4fea9e2a-77c0-48c4-8517-3e104fc043b8" (UID: "4fea9e2a-77c0-48c4-8517-3e104fc043b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:34.128805 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.128765    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/359c47fc-f8af-43f8-865b-1f187fb2672c-istio-token" pod "359c47fc-f8af-43f8-865b-1f187fb2672c" (UID: "359c47fc-f8af-43f8-865b-1f187fb2672c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:34.130526 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.129051    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/359c47fc-f8af-43f8-865b-1f187fb2672c-credential-socket" pod "359c47fc-f8af-43f8-865b-1f187fb2672c" (UID: "359c47fc-f8af-43f8-865b-1f187fb2672c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:34.132932 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.132909    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9"
Oct 30 19:37:34.133054 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.133025    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fea9e2a-77c0-48c4-8517-3e104fc043b8-istio-envoy" pod "4fea9e2a-77c0-48c4-8517-3e104fc043b8" (UID: "4fea9e2a-77c0-48c4-8517-3e104fc043b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:34.135086 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.135057    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/359c47fc-f8af-43f8-865b-1f187fb2672c-istio-data" pod "359c47fc-f8af-43f8-865b-1f187fb2672c" (UID: "359c47fc-f8af-43f8-865b-1f187fb2672c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:34.135174 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.135153    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fea9e2a-77c0-48c4-8517-3e104fc043b8-istio-token" pod "4fea9e2a-77c0-48c4-8517-3e104fc043b8" (UID: "4fea9e2a-77c0-48c4-8517-3e104fc043b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:34.135404 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.135379    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/359c47fc-f8af-43f8-865b-1f187fb2672c-istiod-ca-cert" pod "359c47fc-f8af-43f8-865b-1f187fb2672c" (UID: "359c47fc-f8af-43f8-865b-1f187fb2672c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:37:34.135654 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.135631    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/359c47fc-f8af-43f8-865b-1f187fb2672c-workload-certs" pod "359c47fc-f8af-43f8-865b-1f187fb2672c" (UID: "359c47fc-f8af-43f8-865b-1f187fb2672c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:34.155976 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.155604    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4fea9e2a-77c0-48c4-8517-3e104fc043b8-istio-podinfo" pod "4fea9e2a-77c0-48c4-8517-3e104fc043b8" (UID: "4fea9e2a-77c0-48c4-8517-3e104fc043b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:37:34.171688 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.171632    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/359c47fc-f8af-43f8-865b-1f187fb2672c-istio-podinfo" pod "359c47fc-f8af-43f8-865b-1f187fb2672c" (UID: "359c47fc-f8af-43f8-865b-1f187fb2672c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:37:34.179879 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.172347    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/359c47fc-f8af-43f8-865b-1f187fb2672c-kube-api-access-989kw" pod "359c47fc-f8af-43f8-865b-1f187fb2672c" (UID: "359c47fc-f8af-43f8-865b-1f187fb2672c"). InnerVolumeSpecName "kube-api-access-989kw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:34.179879 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.179010    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg"
Oct 30 19:37:34.182530 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.181903    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fea9e2a-77c0-48c4-8517-3e104fc043b8-kube-api-access-mzbfz" pod "4fea9e2a-77c0-48c4-8517-3e104fc043b8" (UID: "4fea9e2a-77c0-48c4-8517-3e104fc043b8"). InnerVolumeSpecName "kube-api-access-mzbfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:34.184792 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.184539    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j"
Oct 30 19:37:34.186820 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.186129    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk" podStartSLOduration=3.186112849 podStartE2EDuration="3.186112849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:33.260446931 +0000 UTC m=+1272.771506897" watchObservedRunningTime="2026-10-30 19:37:34.186112849 +0000 UTC m=+1273.697172814"
Oct 30 19:37:34.196021 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.195886    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q"
Oct 30 19:37:34.196523 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.196432    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/359c47fc-f8af-43f8-865b-1f187fb2672c-istio-envoy" pod "359c47fc-f8af-43f8-865b-1f187fb2672c" (UID: "359c47fc-f8af-43f8-865b-1f187fb2672c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:34.215626 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.214801    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a851569d-8aa4-4af2-9f3c-c90c4895b95a-istio-token\" (UniqueName: \"kubernetes.io/projected/a851569d-8aa4-4af2-9f3c-c90c4895b95a-istio-token\") pod \"a851569d-8aa4-4af2-9f3c-c90c4895b95a\" (UID: \"a851569d-8aa4-4af2-9f3c-c90c4895b95a\") "
Oct 30 19:37:34.215626 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.214859    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a851569d-8aa4-4af2-9f3c-c90c4895b95a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a851569d-8aa4-4af2-9f3c-c90c4895b95a-istiod-ca-cert\") pod \"a851569d-8aa4-4af2-9f3c-c90c4895b95a\" (UID: \"a851569d-8aa4-4af2-9f3c-c90c4895b95a\") "
Oct 30 19:37:34.215626 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.214902    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a851569d-8aa4-4af2-9f3c-c90c4895b95a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a851569d-8aa4-4af2-9f3c-c90c4895b95a-credential-socket\") pod \"a851569d-8aa4-4af2-9f3c-c90c4895b95a\" (UID: \"a851569d-8aa4-4af2-9f3c-c90c4895b95a\") "
Oct 30 19:37:34.215626 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.214942    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-istio-data\") pod \"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c\" (UID: \"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c\") "
Oct 30 19:37:34.215626 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.214974    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/224648b8-553f-44ad-9d9a-3784a7bea898-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/224648b8-553f-44ad-9d9a-3784a7bea898-credential-socket\") pod \"224648b8-553f-44ad-9d9a-3784a7bea898\" (UID: \"224648b8-553f-44ad-9d9a-3784a7bea898\") "
Oct 30 19:37:34.215626 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.215026    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/224648b8-553f-44ad-9d9a-3784a7bea898-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/224648b8-553f-44ad-9d9a-3784a7bea898-istiod-ca-cert\") pod \"224648b8-553f-44ad-9d9a-3784a7bea898\" (UID: \"224648b8-553f-44ad-9d9a-3784a7bea898\") "
Oct 30 19:37:34.215626 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.215070    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-workload-socket\") pod \"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c\" (UID: \"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c\") "
Oct 30 19:37:34.215626 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.215110    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-kube-api-access-qvglx\" (UniqueName: \"kubernetes.io/projected/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-kube-api-access-qvglx\") pod \"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c\" (UID: \"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c\") "
Oct 30 19:37:34.215626 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.215151    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8a7200d-6acf-4008-8156-d9102730ac76-kube-api-access-7h9j8\" (UniqueName: \"kubernetes.io/projected/d8a7200d-6acf-4008-8156-d9102730ac76-kube-api-access-7h9j8\") pod \"d8a7200d-6acf-4008-8156-d9102730ac76\" (UID: \"d8a7200d-6acf-4008-8156-d9102730ac76\") "
Oct 30 19:37:34.215626 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.215202    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/224648b8-553f-44ad-9d9a-3784a7bea898-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/224648b8-553f-44ad-9d9a-3784a7bea898-workload-certs\") pod \"224648b8-553f-44ad-9d9a-3784a7bea898\" (UID: \"224648b8-553f-44ad-9d9a-3784a7bea898\") "
Oct 30 19:37:34.215626 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.215228    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8a7200d-6acf-4008-8156-d9102730ac76-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8a7200d-6acf-4008-8156-d9102730ac76-istiod-ca-cert\") pod \"d8a7200d-6acf-4008-8156-d9102730ac76\" (UID: \"d8a7200d-6acf-4008-8156-d9102730ac76\") "
Oct 30 19:37:34.215626 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.215259    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a851569d-8aa4-4af2-9f3c-c90c4895b95a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a851569d-8aa4-4af2-9f3c-c90c4895b95a-istio-podinfo\") pod \"a851569d-8aa4-4af2-9f3c-c90c4895b95a\" (UID: \"a851569d-8aa4-4af2-9f3c-c90c4895b95a\") "
Oct 30 19:37:34.215626 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.215287    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8a7200d-6acf-4008-8156-d9102730ac76-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8a7200d-6acf-4008-8156-d9102730ac76-istio-envoy\") pod \"d8a7200d-6acf-4008-8156-d9102730ac76\" (UID: \"d8a7200d-6acf-4008-8156-d9102730ac76\") "
Oct 30 19:37:34.215626 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.215324    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a851569d-8aa4-4af2-9f3c-c90c4895b95a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a851569d-8aa4-4af2-9f3c-c90c4895b95a-istio-data\") pod \"a851569d-8aa4-4af2-9f3c-c90c4895b95a\" (UID: \"a851569d-8aa4-4af2-9f3c-c90c4895b95a\") "
Oct 30 19:37:34.215626 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.215387    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a851569d-8aa4-4af2-9f3c-c90c4895b95a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a851569d-8aa4-4af2-9f3c-c90c4895b95a-workload-socket\") pod \"a851569d-8aa4-4af2-9f3c-c90c4895b95a\" (UID: \"a851569d-8aa4-4af2-9f3c-c90c4895b95a\") "
Oct 30 19:37:34.215626 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.215414    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8a7200d-6acf-4008-8156-d9102730ac76-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8a7200d-6acf-4008-8156-d9102730ac76-istio-data\") pod \"d8a7200d-6acf-4008-8156-d9102730ac76\" (UID: \"d8a7200d-6acf-4008-8156-d9102730ac76\") "
Oct 30 19:37:34.215626 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.215449    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8a7200d-6acf-4008-8156-d9102730ac76-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8a7200d-6acf-4008-8156-d9102730ac76-workload-certs\") pod \"d8a7200d-6acf-4008-8156-d9102730ac76\" (UID: \"d8a7200d-6acf-4008-8156-d9102730ac76\") "
Oct 30 19:37:34.215626 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.215515    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/224648b8-553f-44ad-9d9a-3784a7bea898-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/224648b8-553f-44ad-9d9a-3784a7bea898-istio-envoy\") pod \"224648b8-553f-44ad-9d9a-3784a7bea898\" (UID: \"224648b8-553f-44ad-9d9a-3784a7bea898\") "
Oct 30 19:37:34.215626 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.215544    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-credential-socket\") pod \"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c\" (UID: \"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c\") "
Oct 30 19:37:34.215626 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.215626    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/224648b8-553f-44ad-9d9a-3784a7bea898-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/224648b8-553f-44ad-9d9a-3784a7bea898-workload-socket\") pod \"224648b8-553f-44ad-9d9a-3784a7bea898\" (UID: \"224648b8-553f-44ad-9d9a-3784a7bea898\") "
Oct 30 19:37:34.216845 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.215665    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-istio-envoy\") pod \"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c\" (UID: \"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c\") "
Oct 30 19:37:34.216845 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.215699    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/224648b8-553f-44ad-9d9a-3784a7bea898-istio-token\" (UniqueName: \"kubernetes.io/projected/224648b8-553f-44ad-9d9a-3784a7bea898-istio-token\") pod \"224648b8-553f-44ad-9d9a-3784a7bea898\" (UID: \"224648b8-553f-44ad-9d9a-3784a7bea898\") "
Oct 30 19:37:34.216845 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.215763    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a851569d-8aa4-4af2-9f3c-c90c4895b95a-kube-api-access-h89m8\" (UniqueName: \"kubernetes.io/projected/a851569d-8aa4-4af2-9f3c-c90c4895b95a-kube-api-access-h89m8\") pod \"a851569d-8aa4-4af2-9f3c-c90c4895b95a\" (UID: \"a851569d-8aa4-4af2-9f3c-c90c4895b95a\") "
Oct 30 19:37:34.216845 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.215800    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8a7200d-6acf-4008-8156-d9102730ac76-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a7200d-6acf-4008-8156-d9102730ac76-workload-socket\") pod \"d8a7200d-6acf-4008-8156-d9102730ac76\" (UID: \"d8a7200d-6acf-4008-8156-d9102730ac76\") "
Oct 30 19:37:34.216845 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.215836    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-istio-token\" (UniqueName: \"kubernetes.io/projected/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-istio-token\") pod \"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c\" (UID: \"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c\") "
Oct 30 19:37:34.216845 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.215873    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a851569d-8aa4-4af2-9f3c-c90c4895b95a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a851569d-8aa4-4af2-9f3c-c90c4895b95a-workload-certs\") pod \"a851569d-8aa4-4af2-9f3c-c90c4895b95a\" (UID: \"a851569d-8aa4-4af2-9f3c-c90c4895b95a\") "
Oct 30 19:37:34.216845 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.215905    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-istio-podinfo\") pod \"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c\" (UID: \"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c\") "
Oct 30 19:37:34.216845 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.215956    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8a7200d-6acf-4008-8156-d9102730ac76-istio-token\" (UniqueName: \"kubernetes.io/projected/d8a7200d-6acf-4008-8156-d9102730ac76-istio-token\") pod \"d8a7200d-6acf-4008-8156-d9102730ac76\" (UID: \"d8a7200d-6acf-4008-8156-d9102730ac76\") "
Oct 30 19:37:34.216845 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.216063    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/224648b8-553f-44ad-9d9a-3784a7bea898-kube-api-access-fqbx6\" (UniqueName: \"kubernetes.io/projected/224648b8-553f-44ad-9d9a-3784a7bea898-kube-api-access-fqbx6\") pod \"224648b8-553f-44ad-9d9a-3784a7bea898\" (UID: \"224648b8-553f-44ad-9d9a-3784a7bea898\") "
Oct 30 19:37:34.216845 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.216121    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/224648b8-553f-44ad-9d9a-3784a7bea898-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/224648b8-553f-44ad-9d9a-3784a7bea898-istio-podinfo\") pod \"224648b8-553f-44ad-9d9a-3784a7bea898\" (UID: \"224648b8-553f-44ad-9d9a-3784a7bea898\") "
Oct 30 19:37:34.216845 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.216160    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8a7200d-6acf-4008-8156-d9102730ac76-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8a7200d-6acf-4008-8156-d9102730ac76-istio-podinfo\") pod \"d8a7200d-6acf-4008-8156-d9102730ac76\" (UID: \"d8a7200d-6acf-4008-8156-d9102730ac76\") "
Oct 30 19:37:34.216845 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.216416    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-workload-certs\") pod \"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c\" (UID: \"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c\") "
Oct 30 19:37:34.216845 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.216499    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a851569d-8aa4-4af2-9f3c-c90c4895b95a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a851569d-8aa4-4af2-9f3c-c90c4895b95a-istio-envoy\") pod \"a851569d-8aa4-4af2-9f3c-c90c4895b95a\" (UID: \"a851569d-8aa4-4af2-9f3c-c90c4895b95a\") "
Oct 30 19:37:34.216845 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.216540    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8a7200d-6acf-4008-8156-d9102730ac76-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a7200d-6acf-4008-8156-d9102730ac76-credential-socket\") pod \"d8a7200d-6acf-4008-8156-d9102730ac76\" (UID: \"d8a7200d-6acf-4008-8156-d9102730ac76\") "
Oct 30 19:37:34.216845 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.216606    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-istiod-ca-cert\") pod \"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c\" (UID: \"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c\") "
Oct 30 19:37:34.216845 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.216655    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/224648b8-553f-44ad-9d9a-3784a7bea898-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/224648b8-553f-44ad-9d9a-3784a7bea898-istio-data\") pod \"224648b8-553f-44ad-9d9a-3784a7bea898\" (UID: \"224648b8-553f-44ad-9d9a-3784a7bea898\") "
Oct 30 19:37:34.217713 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.217690    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fea9e2a-77c0-48c4-8517-3e104fc043b8-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.217830 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.217719    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/359c47fc-f8af-43f8-865b-1f187fb2672c-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.217830 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.217754    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzbfz\" (UniqueName: \"kubernetes.io/projected/4fea9e2a-77c0-48c4-8517-3e104fc043b8-kube-api-access-mzbfz\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.217830 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.217769    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fea9e2a-77c0-48c4-8517-3e104fc043b8-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.217830 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.217783    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/359c47fc-f8af-43f8-865b-1f187fb2672c-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.217830 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.217799    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/359c47fc-f8af-43f8-865b-1f187fb2672c-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.217830 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.217814    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/359c47fc-f8af-43f8-865b-1f187fb2672c-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.217830 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.217828    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fea9e2a-77c0-48c4-8517-3e104fc043b8-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.218278 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.217841    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/359c47fc-f8af-43f8-865b-1f187fb2672c-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.218278 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.217858    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-989kw\" (UniqueName: \"kubernetes.io/projected/359c47fc-f8af-43f8-865b-1f187fb2672c-kube-api-access-989kw\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.218278 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.217871    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/359c47fc-f8af-43f8-865b-1f187fb2672c-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.218278 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.217885    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/359c47fc-f8af-43f8-865b-1f187fb2672c-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.218278 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.217898    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fea9e2a-77c0-48c4-8517-3e104fc043b8-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.218278 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.217912    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fea9e2a-77c0-48c4-8517-3e104fc043b8-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.218278 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.217926    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/359c47fc-f8af-43f8-865b-1f187fb2672c-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.218278 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.217940    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fea9e2a-77c0-48c4-8517-3e104fc043b8-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.218278 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.217955    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fea9e2a-77c0-48c4-8517-3e104fc043b8-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.218278 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.217968    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fea9e2a-77c0-48c4-8517-3e104fc043b8-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.221251 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.220425    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8a7200d-6acf-4008-8156-d9102730ac76-workload-certs" pod "d8a7200d-6acf-4008-8156-d9102730ac76" (UID: "d8a7200d-6acf-4008-8156-d9102730ac76"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:34.221251 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.220795    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a851569d-8aa4-4af2-9f3c-c90c4895b95a-istio-data" pod "a851569d-8aa4-4af2-9f3c-c90c4895b95a" (UID: "a851569d-8aa4-4af2-9f3c-c90c4895b95a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:34.221251 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.220983    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8a7200d-6acf-4008-8156-d9102730ac76-istio-data" pod "d8a7200d-6acf-4008-8156-d9102730ac76" (UID: "d8a7200d-6acf-4008-8156-d9102730ac76"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:34.221522 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.221311    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a851569d-8aa4-4af2-9f3c-c90c4895b95a-workload-socket" pod "a851569d-8aa4-4af2-9f3c-c90c4895b95a" (UID: "a851569d-8aa4-4af2-9f3c-c90c4895b95a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:34.221671 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.221649    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8a7200d-6acf-4008-8156-d9102730ac76-istiod-ca-cert" pod "d8a7200d-6acf-4008-8156-d9102730ac76" (UID: "d8a7200d-6acf-4008-8156-d9102730ac76"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:37:34.222024 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.221964    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/224648b8-553f-44ad-9d9a-3784a7bea898-workload-socket" pod "224648b8-553f-44ad-9d9a-3784a7bea898" (UID: "224648b8-553f-44ad-9d9a-3784a7bea898"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:34.222261 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.222234    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/224648b8-553f-44ad-9d9a-3784a7bea898-workload-certs" pod "224648b8-553f-44ad-9d9a-3784a7bea898" (UID: "224648b8-553f-44ad-9d9a-3784a7bea898"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:34.222651 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.222628    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a851569d-8aa4-4af2-9f3c-c90c4895b95a-istiod-ca-cert" pod "a851569d-8aa4-4af2-9f3c-c90c4895b95a" (UID: "a851569d-8aa4-4af2-9f3c-c90c4895b95a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:37:34.222910 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.222887    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a851569d-8aa4-4af2-9f3c-c90c4895b95a-credential-socket" pod "a851569d-8aa4-4af2-9f3c-c90c4895b95a" (UID: "a851569d-8aa4-4af2-9f3c-c90c4895b95a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:34.223103 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.223080    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-istio-data" pod "c3185a2a-d994-41d4-a8d7-9b7e966b9f1c" (UID: "c3185a2a-d994-41d4-a8d7-9b7e966b9f1c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:34.223272 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.223253    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/224648b8-553f-44ad-9d9a-3784a7bea898-credential-socket" pod "224648b8-553f-44ad-9d9a-3784a7bea898" (UID: "224648b8-553f-44ad-9d9a-3784a7bea898"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:34.223518 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.223497    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/224648b8-553f-44ad-9d9a-3784a7bea898-istiod-ca-cert" pod "224648b8-553f-44ad-9d9a-3784a7bea898" (UID: "224648b8-553f-44ad-9d9a-3784a7bea898"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:37:34.223819 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.223797    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-workload-socket" pod "c3185a2a-d994-41d4-a8d7-9b7e966b9f1c" (UID: "c3185a2a-d994-41d4-a8d7-9b7e966b9f1c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:34.224447 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.224352    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-credential-socket" pod "c3185a2a-d994-41d4-a8d7-9b7e966b9f1c" (UID: "c3185a2a-d994-41d4-a8d7-9b7e966b9f1c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:34.250799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.233323    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a851569d-8aa4-4af2-9f3c-c90c4895b95a-istio-podinfo" pod "a851569d-8aa4-4af2-9f3c-c90c4895b95a" (UID: "a851569d-8aa4-4af2-9f3c-c90c4895b95a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:37:34.250799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.233559    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-workload-certs" pod "c3185a2a-d994-41d4-a8d7-9b7e966b9f1c" (UID: "c3185a2a-d994-41d4-a8d7-9b7e966b9f1c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:34.250799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.234193    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/224648b8-553f-44ad-9d9a-3784a7bea898-istio-envoy" pod "224648b8-553f-44ad-9d9a-3784a7bea898" (UID: "224648b8-553f-44ad-9d9a-3784a7bea898"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:34.250799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.234398    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8a7200d-6acf-4008-8156-d9102730ac76-credential-socket" pod "d8a7200d-6acf-4008-8156-d9102730ac76" (UID: "d8a7200d-6acf-4008-8156-d9102730ac76"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:34.250799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.234765    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8a7200d-6acf-4008-8156-d9102730ac76-istio-envoy" pod "d8a7200d-6acf-4008-8156-d9102730ac76" (UID: "d8a7200d-6acf-4008-8156-d9102730ac76"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:34.250799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.235004    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-kube-api-access-qvglx" pod "c3185a2a-d994-41d4-a8d7-9b7e966b9f1c" (UID: "c3185a2a-d994-41d4-a8d7-9b7e966b9f1c"). InnerVolumeSpecName "kube-api-access-qvglx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:34.250799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.245867    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/224648b8-553f-44ad-9d9a-3784a7bea898-istio-data" pod "224648b8-553f-44ad-9d9a-3784a7bea898" (UID: "224648b8-553f-44ad-9d9a-3784a7bea898"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:34.250799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.245937    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8a7200d-6acf-4008-8156-d9102730ac76-workload-socket" pod "d8a7200d-6acf-4008-8156-d9102730ac76" (UID: "d8a7200d-6acf-4008-8156-d9102730ac76"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:34.250799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.245985    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a851569d-8aa4-4af2-9f3c-c90c4895b95a-workload-certs" pod "a851569d-8aa4-4af2-9f3c-c90c4895b95a" (UID: "a851569d-8aa4-4af2-9f3c-c90c4895b95a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:34.267701 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.267603    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-istiod-ca-cert" pod "c3185a2a-d994-41d4-a8d7-9b7e966b9f1c" (UID: "c3185a2a-d994-41d4-a8d7-9b7e966b9f1c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:37:34.268075 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.268026    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/224648b8-553f-44ad-9d9a-3784a7bea898-istio-token" pod "224648b8-553f-44ad-9d9a-3784a7bea898" (UID: "224648b8-553f-44ad-9d9a-3784a7bea898"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:34.273775 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.272340    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/224648b8-553f-44ad-9d9a-3784a7bea898-istio-podinfo" pod "224648b8-553f-44ad-9d9a-3784a7bea898" (UID: "224648b8-553f-44ad-9d9a-3784a7bea898"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:37:34.313246 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.302749    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8a7200d-6acf-4008-8156-d9102730ac76-istio-podinfo" pod "d8a7200d-6acf-4008-8156-d9102730ac76" (UID: "d8a7200d-6acf-4008-8156-d9102730ac76"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:37:34.313246 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.309795    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-istio-podinfo" pod "c3185a2a-d994-41d4-a8d7-9b7e966b9f1c" (UID: "c3185a2a-d994-41d4-a8d7-9b7e966b9f1c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:37:34.313246 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.311001    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a851569d-8aa4-4af2-9f3c-c90c4895b95a-kube-api-access-h89m8" pod "a851569d-8aa4-4af2-9f3c-c90c4895b95a" (UID: "a851569d-8aa4-4af2-9f3c-c90c4895b95a"). InnerVolumeSpecName "kube-api-access-h89m8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:34.333036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.332561    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a851569d-8aa4-4af2-9f3c-c90c4895b95a-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.333036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.332596    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a851569d-8aa4-4af2-9f3c-c90c4895b95a-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.333036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.332610    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.333036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.332628    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/224648b8-553f-44ad-9d9a-3784a7bea898-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.333036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.332643    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/224648b8-553f-44ad-9d9a-3784a7bea898-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.333036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.332658    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.333036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.332672    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvglx\" (UniqueName: \"kubernetes.io/projected/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-kube-api-access-qvglx\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.333036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.332685    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/224648b8-553f-44ad-9d9a-3784a7bea898-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.333036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.332699    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8a7200d-6acf-4008-8156-d9102730ac76-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.333036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.332712    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a851569d-8aa4-4af2-9f3c-c90c4895b95a-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.333036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.332728    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8a7200d-6acf-4008-8156-d9102730ac76-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.333036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.332770    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a851569d-8aa4-4af2-9f3c-c90c4895b95a-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.333036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.332783    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a851569d-8aa4-4af2-9f3c-c90c4895b95a-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.333036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.332795    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8a7200d-6acf-4008-8156-d9102730ac76-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.333036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.332812    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8a7200d-6acf-4008-8156-d9102730ac76-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.333036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.332829    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/224648b8-553f-44ad-9d9a-3784a7bea898-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.333036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.332840    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.333036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.332853    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/224648b8-553f-44ad-9d9a-3784a7bea898-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.333036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.332867    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/224648b8-553f-44ad-9d9a-3784a7bea898-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.333036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.332881    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h89m8\" (UniqueName: \"kubernetes.io/projected/a851569d-8aa4-4af2-9f3c-c90c4895b95a-kube-api-access-h89m8\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.333036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.332895    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a7200d-6acf-4008-8156-d9102730ac76-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.333036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.332908    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a851569d-8aa4-4af2-9f3c-c90c4895b95a-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.333036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.332921    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.333036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.332934    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/224648b8-553f-44ad-9d9a-3784a7bea898-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.333036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.332947    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8a7200d-6acf-4008-8156-d9102730ac76-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.333036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.332960    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.333036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.332973    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a7200d-6acf-4008-8156-d9102730ac76-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.333036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.332986    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.333036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.332999    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/224648b8-553f-44ad-9d9a-3784a7bea898-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.336953 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.336829    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a851569d-8aa4-4af2-9f3c-c90c4895b95a-istio-token" pod "a851569d-8aa4-4af2-9f3c-c90c4895b95a" (UID: "a851569d-8aa4-4af2-9f3c-c90c4895b95a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:34.336953 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.336918    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8a7200d-6acf-4008-8156-d9102730ac76-istio-token" pod "d8a7200d-6acf-4008-8156-d9102730ac76" (UID: "d8a7200d-6acf-4008-8156-d9102730ac76"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:34.369916 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.351909    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8a7200d-6acf-4008-8156-d9102730ac76-kube-api-access-7h9j8" pod "d8a7200d-6acf-4008-8156-d9102730ac76" (UID: "d8a7200d-6acf-4008-8156-d9102730ac76"). InnerVolumeSpecName "kube-api-access-7h9j8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:34.369916 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.352133    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-istio-token" pod "c3185a2a-d994-41d4-a8d7-9b7e966b9f1c" (UID: "c3185a2a-d994-41d4-a8d7-9b7e966b9f1c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:34.451560 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.439050    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-istio-envoy" pod "c3185a2a-d994-41d4-a8d7-9b7e966b9f1c" (UID: "c3185a2a-d994-41d4-a8d7-9b7e966b9f1c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:34.451560 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.439200    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/224648b8-553f-44ad-9d9a-3784a7bea898-kube-api-access-fqbx6" pod "224648b8-553f-44ad-9d9a-3784a7bea898" (UID: "224648b8-553f-44ad-9d9a-3784a7bea898"). InnerVolumeSpecName "kube-api-access-fqbx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:34.465948 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.450855    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a851569d-8aa4-4af2-9f3c-c90c4895b95a-istio-envoy" pod "a851569d-8aa4-4af2-9f3c-c90c4895b95a" (UID: "a851569d-8aa4-4af2-9f3c-c90c4895b95a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:34.491761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.491701    5043 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Oct 30 19:37:34.492023 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.491998    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk" podUID="6ae382e3-be21-4e69-bf70-a8c0b8517789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Oct 30 19:37:34.507571 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.503178    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.507836 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.507817    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.507966 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.507953    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8a7200d-6acf-4008-8156-d9102730ac76-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.510769 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.508049    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqbx6\" (UniqueName: \"kubernetes.io/projected/224648b8-553f-44ad-9d9a-3784a7bea898-kube-api-access-fqbx6\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.510769 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.508069    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a851569d-8aa4-4af2-9f3c-c90c4895b95a-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.510769 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.508158    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a851569d-8aa4-4af2-9f3c-c90c4895b95a-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.510769 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.508171    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h9j8\" (UniqueName: \"kubernetes.io/projected/d8a7200d-6acf-4008-8156-d9102730ac76-kube-api-access-7h9j8\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.545501 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.545081    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9" event={"ID":"224648b8-553f-44ad-9d9a-3784a7bea898","Type":"ContainerDied","Data":"02dac12cdbc43ead9ac8efc1a34ab807df13b544beb31d3a04ff3bb0138e90b3"}
Oct 30 19:37:34.545776 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.545759    5043 scope.go:122] "RemoveContainer" containerID="4ea546a3bab408e0b58ecbeeab9f86306a7f29763425dcad8cdf95930b5cb1a4"
Oct 30 19:37:34.545992 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.545189    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9"
Oct 30 19:37:34.547857 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.547838    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7"
Oct 30 19:37:34.551102 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.551078    5043 generic.go:356] "Generic (PLEG): container finished" podID="8ce15b3c-1548-489c-a4f1-2fc151a53d35" containerID="5680ee690c20c4258226d6037905090e6336e4a1a7f3ec1f7c9c35d6924fd60c" exitCode=0
Oct 30 19:37:34.551221 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.551187    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7" event={"ID":"8ce15b3c-1548-489c-a4f1-2fc151a53d35","Type":"ContainerDied","Data":"5680ee690c20c4258226d6037905090e6336e4a1a7f3ec1f7c9c35d6924fd60c"}
Oct 30 19:37:34.551293 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.551228    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7" event={"ID":"8ce15b3c-1548-489c-a4f1-2fc151a53d35","Type":"ContainerDied","Data":"67365e585d64041578240b803b50091d245fdce4fe0fd824b12fc55db87936b6"}
Oct 30 19:37:34.553000 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.552973    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j" event={"ID":"c3185a2a-d994-41d4-a8d7-9b7e966b9f1c","Type":"ContainerDied","Data":"99a241c37b6357bab02d273b53fa120ad8c1b5bb46292bda0db2b06e2879d92a"}
Oct 30 19:37:34.553423 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.553405    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j"
Oct 30 19:37:34.556591 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.556573    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf"
Oct 30 19:37:34.557555 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.557533    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf" event={"ID":"4fea9e2a-77c0-48c4-8517-3e104fc043b8","Type":"ContainerDied","Data":"90b6d022465c3074cc0ac581da9058c50a0271f15227578f3b1e0ba5dc9f1a92"}
Oct 30 19:37:34.563550 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.563473    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q" event={"ID":"d8a7200d-6acf-4008-8156-d9102730ac76","Type":"ContainerDied","Data":"a8e2500cf7b15c833686ab1c4bcd6f952d5d308771e5abc4986413fb6253fbdf"}
Oct 30 19:37:34.563795 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.563778    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q"
Oct 30 19:37:34.570874 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.569473    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg" event={"ID":"a851569d-8aa4-4af2-9f3c-c90c4895b95a","Type":"ContainerDied","Data":"693b2f6b8ec1a093329a6ba6da4226c1bdf082a8018f608599260933a86a1527"}
Oct 30 19:37:34.570874 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.569569    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg"
Oct 30 19:37:34.604273 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.603555    5043 scope.go:122] "RemoveContainer" containerID="5680ee690c20c4258226d6037905090e6336e4a1a7f3ec1f7c9c35d6924fd60c"
Oct 30 19:37:34.617817 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.608280    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt"
Oct 30 19:37:34.617817 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.611290    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt" event={"ID":"359c47fc-f8af-43f8-865b-1f187fb2672c","Type":"ContainerDied","Data":"5ede5bedc56af57330642fe3b9543a93e621fe7d4181c547b561aa8228402ca8"}
Oct 30 19:37:34.629709 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.629685    5043 scope.go:122] "RemoveContainer" containerID="5680ee690c20c4258226d6037905090e6336e4a1a7f3ec1f7c9c35d6924fd60c"
Oct 30 19:37:34.630993 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:37:34.630968    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5680ee690c20c4258226d6037905090e6336e4a1a7f3ec1f7c9c35d6924fd60c\": container with ID starting with 5680ee690c20c4258226d6037905090e6336e4a1a7f3ec1f7c9c35d6924fd60c not found: ID does not exist" containerID="5680ee690c20c4258226d6037905090e6336e4a1a7f3ec1f7c9c35d6924fd60c"
Oct 30 19:37:34.631143 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.631119    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5680ee690c20c4258226d6037905090e6336e4a1a7f3ec1f7c9c35d6924fd60c"} err="failed to get container status \"5680ee690c20c4258226d6037905090e6336e4a1a7f3ec1f7c9c35d6924fd60c\": rpc error: code = NotFound desc = could not find container \"5680ee690c20c4258226d6037905090e6336e4a1a7f3ec1f7c9c35d6924fd60c\": container with ID starting with 5680ee690c20c4258226d6037905090e6336e4a1a7f3ec1f7c9c35d6924fd60c not found: ID does not exist"
Oct 30 19:37:34.631251 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.631241    5043 scope.go:122] "RemoveContainer" containerID="20e7692b745948d12f8044909a2acb7baae7326b2556c005133d80f20e87bf91"
Oct 30 19:37:34.715457 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.715391    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ce15b3c-1548-489c-a4f1-2fc151a53d35-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce15b3c-1548-489c-a4f1-2fc151a53d35-credential-socket\") pod \"8ce15b3c-1548-489c-a4f1-2fc151a53d35\" (UID: \"8ce15b3c-1548-489c-a4f1-2fc151a53d35\") "
Oct 30 19:37:34.715924 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.715885    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ce15b3c-1548-489c-a4f1-2fc151a53d35-kube-api-access-4c7sf\" (UniqueName: \"kubernetes.io/projected/8ce15b3c-1548-489c-a4f1-2fc151a53d35-kube-api-access-4c7sf\") pod \"8ce15b3c-1548-489c-a4f1-2fc151a53d35\" (UID: \"8ce15b3c-1548-489c-a4f1-2fc151a53d35\") "
Oct 30 19:37:34.716180 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.716158    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ce15b3c-1548-489c-a4f1-2fc151a53d35-istio-token\" (UniqueName: \"kubernetes.io/projected/8ce15b3c-1548-489c-a4f1-2fc151a53d35-istio-token\") pod \"8ce15b3c-1548-489c-a4f1-2fc151a53d35\" (UID: \"8ce15b3c-1548-489c-a4f1-2fc151a53d35\") "
Oct 30 19:37:34.716430 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.716411    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ce15b3c-1548-489c-a4f1-2fc151a53d35-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ce15b3c-1548-489c-a4f1-2fc151a53d35-istio-podinfo\") pod \"8ce15b3c-1548-489c-a4f1-2fc151a53d35\" (UID: \"8ce15b3c-1548-489c-a4f1-2fc151a53d35\") "
Oct 30 19:37:34.716667 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.716645    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ce15b3c-1548-489c-a4f1-2fc151a53d35-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ce15b3c-1548-489c-a4f1-2fc151a53d35-istiod-ca-cert\") pod \"8ce15b3c-1548-489c-a4f1-2fc151a53d35\" (UID: \"8ce15b3c-1548-489c-a4f1-2fc151a53d35\") "
Oct 30 19:37:34.716945 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.716924    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ce15b3c-1548-489c-a4f1-2fc151a53d35-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce15b3c-1548-489c-a4f1-2fc151a53d35-workload-socket\") pod \"8ce15b3c-1548-489c-a4f1-2fc151a53d35\" (UID: \"8ce15b3c-1548-489c-a4f1-2fc151a53d35\") "
Oct 30 19:37:34.717136 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.717120    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ce15b3c-1548-489c-a4f1-2fc151a53d35-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ce15b3c-1548-489c-a4f1-2fc151a53d35-istio-envoy\") pod \"8ce15b3c-1548-489c-a4f1-2fc151a53d35\" (UID: \"8ce15b3c-1548-489c-a4f1-2fc151a53d35\") "
Oct 30 19:37:34.717320 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.717304    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ce15b3c-1548-489c-a4f1-2fc151a53d35-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ce15b3c-1548-489c-a4f1-2fc151a53d35-workload-certs\") pod \"8ce15b3c-1548-489c-a4f1-2fc151a53d35\" (UID: \"8ce15b3c-1548-489c-a4f1-2fc151a53d35\") "
Oct 30 19:37:34.717468 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.717452    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ce15b3c-1548-489c-a4f1-2fc151a53d35-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ce15b3c-1548-489c-a4f1-2fc151a53d35-istio-data\") pod \"8ce15b3c-1548-489c-a4f1-2fc151a53d35\" (UID: \"8ce15b3c-1548-489c-a4f1-2fc151a53d35\") "
Oct 30 19:37:34.719056 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.719034    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce15b3c-1548-489c-a4f1-2fc151a53d35-credential-socket" pod "8ce15b3c-1548-489c-a4f1-2fc151a53d35" (UID: "8ce15b3c-1548-489c-a4f1-2fc151a53d35"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:34.827297 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.826981    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ce15b3c-1548-489c-a4f1-2fc151a53d35-istiod-ca-cert" pod "8ce15b3c-1548-489c-a4f1-2fc151a53d35" (UID: "8ce15b3c-1548-489c-a4f1-2fc151a53d35"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:37:34.838432 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.838254    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ce15b3c-1548-489c-a4f1-2fc151a53d35-istio-podinfo" pod "8ce15b3c-1548-489c-a4f1-2fc151a53d35" (UID: "8ce15b3c-1548-489c-a4f1-2fc151a53d35"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:37:34.838432 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.838379    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ce15b3c-1548-489c-a4f1-2fc151a53d35-istio-token" pod "8ce15b3c-1548-489c-a4f1-2fc151a53d35" (UID: "8ce15b3c-1548-489c-a4f1-2fc151a53d35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:34.838920 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.838892    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce15b3c-1548-489c-a4f1-2fc151a53d35-workload-certs" pod "8ce15b3c-1548-489c-a4f1-2fc151a53d35" (UID: "8ce15b3c-1548-489c-a4f1-2fc151a53d35"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:34.839960 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.839928    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce15b3c-1548-489c-a4f1-2fc151a53d35-workload-socket" pod "8ce15b3c-1548-489c-a4f1-2fc151a53d35" (UID: "8ce15b3c-1548-489c-a4f1-2fc151a53d35"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:34.843302 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.843004    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce15b3c-1548-489c-a4f1-2fc151a53d35-istio-envoy" pod "8ce15b3c-1548-489c-a4f1-2fc151a53d35" (UID: "8ce15b3c-1548-489c-a4f1-2fc151a53d35"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:34.850375 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.846825    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce15b3c-1548-489c-a4f1-2fc151a53d35-istio-data" pod "8ce15b3c-1548-489c-a4f1-2fc151a53d35" (UID: "8ce15b3c-1548-489c-a4f1-2fc151a53d35"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:34.858116 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.857891    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ce15b3c-1548-489c-a4f1-2fc151a53d35-kube-api-access-4c7sf" pod "8ce15b3c-1548-489c-a4f1-2fc151a53d35" (UID: "8ce15b3c-1548-489c-a4f1-2fc151a53d35"). InnerVolumeSpecName "kube-api-access-4c7sf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:34.930724 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.930693    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ce15b3c-1548-489c-a4f1-2fc151a53d35-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.931037 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.931019    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ce15b3c-1548-489c-a4f1-2fc151a53d35-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.931778 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.931434    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce15b3c-1548-489c-a4f1-2fc151a53d35-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.932897 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.932880    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ce15b3c-1548-489c-a4f1-2fc151a53d35-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.933392 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.933377    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ce15b3c-1548-489c-a4f1-2fc151a53d35-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.934248 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.934230    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ce15b3c-1548-489c-a4f1-2fc151a53d35-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.934359 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.934347    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce15b3c-1548-489c-a4f1-2fc151a53d35-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.934455 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.934443    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c7sf\" (UniqueName: \"kubernetes.io/projected/8ce15b3c-1548-489c-a4f1-2fc151a53d35-kube-api-access-4c7sf\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.934542 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.934532    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ce15b3c-1548-489c-a4f1-2fc151a53d35-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:34.943304 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.941615    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9"]
Oct 30 19:37:34.943304 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.941649    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-q8kz9"]
Oct 30 19:37:34.943304 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.941676    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6266b58-dd5hf"]
Oct 30 19:37:34.943304 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.941713    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt"]
Oct 30 19:37:34.943304 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.941770    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-npzrt"]
Oct 30 19:37:34.943304 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.941792    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-194e54e4-9dd4p"]
Oct 30 19:37:34.943304 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.941820    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk"]
Oct 30 19:37:34.943304 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.941851    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg"]
Oct 30 19:37:34.943304 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.941868    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-lf4jg"]
Oct 30 19:37:34.943304 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.942037    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk" podUID="c08cde99-b619-487f-b34f-3180ba5019be" containerName="istio-proxy" containerID="cri-o://e1fa23bc7e10f805bb6e9542b2743e6d7381d059afc293bd5967426fdbcc01d8" gracePeriod=30
Oct 30 19:37:34.948268 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.948247    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf"]
Oct 30 19:37:34.953224 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:34.953201    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-pjznf"]
Oct 30 19:37:35.100826 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:35.059840    5043 scope.go:122] "RemoveContainer" containerID="a4d973530c2d7a358d00dfa3f48bfbea57b1ba2a3b9dd90f577c1b97cfccf368"
Oct 30 19:37:35.118954 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:35.118925    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j"]
Oct 30 19:37:35.310906 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:35.310879    5043 scope.go:122] "RemoveContainer" containerID="200b1ca9a3dfa73fa6d763f3b5eefb5d256b363ec4d4ae282a751afd3c04377d"
Oct 30 19:37:35.311662 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:35.311643    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-2d82j"]
Oct 30 19:37:35.323626 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:35.323397    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q"]
Oct 30 19:37:35.328962 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:35.328941    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-gsk6q"]
Oct 30 19:37:35.345920 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:35.345900    5043 scope.go:122] "RemoveContainer" containerID="a6b31cdbbfe4e45387fe8fd60975db8818bece18976ce5c683d46c3fd9affd72"
Oct 30 19:37:35.395641 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:35.395620    5043 scope.go:122] "RemoveContainer" containerID="2b329b14627cf06b423a2cbca22183d9d506012c6472357ef35919dc3b4171f4"
Oct 30 19:37:35.444773 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:35.441849    5043 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Oct 30 19:37:35.444773 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:35.441907    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk" podUID="6ae382e3-be21-4e69-bf70-a8c0b8517789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Oct 30 19:37:35.620010 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:35.619971    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6266b58-dd5hf" event={"ID":"87ce476f-991e-4437-a15e-f7410d2392a5","Type":"ContainerStarted","Data":"8ca7c9b0b2a8f7cfa2c9e3beae99541b7ff1f4c738666327e7a1641a01789b75"}
Oct 30 19:37:35.621220 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:35.621197    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7"
Oct 30 19:37:35.624214 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:35.624177    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-194e54e4-9dd4p" event={"ID":"08851e5a-2c32-446b-9ffa-7508ac46cf2c","Type":"ContainerStarted","Data":"1e133cbd305c8246738c23f6d01278ae191fd786c5920f4691c59b872d0afa72"}
Oct 30 19:37:35.680884 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:35.680796    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7"]
Oct 30 19:37:35.693628 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:35.693593    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-p58s7"]
Oct 30 19:37:36.029556 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.029513    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb"]
Oct 30 19:37:36.030666 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.030647    5043 cpu_manager.go:414] "containerMap: removing container" podUID="224648b8-553f-44ad-9d9a-3784a7bea898" containerName="istio-proxy"
Oct 30 19:37:36.030761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.030670    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="224648b8-553f-44ad-9d9a-3784a7bea898" containerName="istio-proxy"
Oct 30 19:37:36.030761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.030687    5043 cpu_manager.go:414] "containerMap: removing container" podUID="c3185a2a-d994-41d4-a8d7-9b7e966b9f1c" containerName="istio-proxy"
Oct 30 19:37:36.030761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.030694    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3185a2a-d994-41d4-a8d7-9b7e966b9f1c" containerName="istio-proxy"
Oct 30 19:37:36.030761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.030716    5043 cpu_manager.go:414] "containerMap: removing container" podUID="224648b8-553f-44ad-9d9a-3784a7bea898" containerName="istio-proxy"
Oct 30 19:37:36.030761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.030723    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="224648b8-553f-44ad-9d9a-3784a7bea898" containerName="istio-proxy"
Oct 30 19:37:36.030761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.030751    5043 cpu_manager.go:414] "containerMap: removing container" podUID="4fea9e2a-77c0-48c4-8517-3e104fc043b8" containerName="istio-proxy"
Oct 30 19:37:36.030761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.030758    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fea9e2a-77c0-48c4-8517-3e104fc043b8" containerName="istio-proxy"
Oct 30 19:37:36.030979 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.030779    5043 cpu_manager.go:414] "containerMap: removing container" podUID="a851569d-8aa4-4af2-9f3c-c90c4895b95a" containerName="istio-proxy"
Oct 30 19:37:36.030979 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.030786    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a851569d-8aa4-4af2-9f3c-c90c4895b95a" containerName="istio-proxy"
Oct 30 19:37:36.030979 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.030793    5043 cpu_manager.go:414] "containerMap: removing container" podUID="d8a7200d-6acf-4008-8156-d9102730ac76" containerName="istio-proxy"
Oct 30 19:37:36.030979 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.030800    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8a7200d-6acf-4008-8156-d9102730ac76" containerName="istio-proxy"
Oct 30 19:37:36.030979 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.030837    5043 cpu_manager.go:414] "containerMap: removing container" podUID="359c47fc-f8af-43f8-865b-1f187fb2672c" containerName="istio-proxy"
Oct 30 19:37:36.030979 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.030843    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="359c47fc-f8af-43f8-865b-1f187fb2672c" containerName="istio-proxy"
Oct 30 19:37:36.030979 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.030860    5043 cpu_manager.go:414] "containerMap: removing container" podUID="8ce15b3c-1548-489c-a4f1-2fc151a53d35" containerName="istio-proxy"
Oct 30 19:37:36.030979 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.030868    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ce15b3c-1548-489c-a4f1-2fc151a53d35" containerName="istio-proxy"
Oct 30 19:37:36.031329 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.031312    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="359c47fc-f8af-43f8-865b-1f187fb2672c" containerName="istio-proxy"
Oct 30 19:37:36.031371 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.031334    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ce15b3c-1548-489c-a4f1-2fc151a53d35" containerName="istio-proxy"
Oct 30 19:37:36.031371 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.031350    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="224648b8-553f-44ad-9d9a-3784a7bea898" containerName="istio-proxy"
Oct 30 19:37:36.031371 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.031357    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3185a2a-d994-41d4-a8d7-9b7e966b9f1c" containerName="istio-proxy"
Oct 30 19:37:36.031371 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.031369    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fea9e2a-77c0-48c4-8517-3e104fc043b8" containerName="istio-proxy"
Oct 30 19:37:36.031481 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.031381    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="a851569d-8aa4-4af2-9f3c-c90c4895b95a" containerName="istio-proxy"
Oct 30 19:37:36.031481 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.031390    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8a7200d-6acf-4008-8156-d9102730ac76" containerName="istio-proxy"
Oct 30 19:37:36.035955 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.035935    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb"
Oct 30 19:37:36.042455 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.041973    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-579ddb-openshift-default-dockercfg-2w4dv\""
Oct 30 19:37:36.107066 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.105779    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb"]
Oct 30 19:37:36.112847 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.112825    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c584699-3a9f-45e1-a380-cb724ff74820-credential-socket\") pod \"e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb\" (UID: \"7c584699-3a9f-45e1-a380-cb724ff74820\") " pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb"
Oct 30 19:37:36.112953 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.112934    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c584699-3a9f-45e1-a380-cb724ff74820-workload-certs\") pod \"e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb\" (UID: \"7c584699-3a9f-45e1-a380-cb724ff74820\") " pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb"
Oct 30 19:37:36.113002 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.112961    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c584699-3a9f-45e1-a380-cb724ff74820-istiod-ca-cert\") pod \"e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb\" (UID: \"7c584699-3a9f-45e1-a380-cb724ff74820\") " pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb"
Oct 30 19:37:36.113083 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.113071    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c584699-3a9f-45e1-a380-cb724ff74820-istio-token\") pod \"e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb\" (UID: \"7c584699-3a9f-45e1-a380-cb724ff74820\") " pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb"
Oct 30 19:37:36.113128 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.113112    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vw4c\" (UniqueName: \"kubernetes.io/projected/7c584699-3a9f-45e1-a380-cb724ff74820-kube-api-access-4vw4c\") pod \"e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb\" (UID: \"7c584699-3a9f-45e1-a380-cb724ff74820\") " pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb"
Oct 30 19:37:36.113170 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.113158    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c584699-3a9f-45e1-a380-cb724ff74820-istio-podinfo\") pod \"e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb\" (UID: \"7c584699-3a9f-45e1-a380-cb724ff74820\") " pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb"
Oct 30 19:37:36.113209 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.113200    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c584699-3a9f-45e1-a380-cb724ff74820-istio-envoy\") pod \"e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb\" (UID: \"7c584699-3a9f-45e1-a380-cb724ff74820\") " pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb"
Oct 30 19:37:36.113252 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.113242    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c584699-3a9f-45e1-a380-cb724ff74820-workload-socket\") pod \"e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb\" (UID: \"7c584699-3a9f-45e1-a380-cb724ff74820\") " pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb"
Oct 30 19:37:36.113285 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.113273    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c584699-3a9f-45e1-a380-cb724ff74820-istio-data\") pod \"e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb\" (UID: \"7c584699-3a9f-45e1-a380-cb724ff74820\") " pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb"
Oct 30 19:37:36.216099 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.215848    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c584699-3a9f-45e1-a380-cb724ff74820-istio-token\") pod \"e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb\" (UID: \"7c584699-3a9f-45e1-a380-cb724ff74820\") " pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb"
Oct 30 19:37:36.216099 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.215934    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vw4c\" (UniqueName: \"kubernetes.io/projected/7c584699-3a9f-45e1-a380-cb724ff74820-kube-api-access-4vw4c\") pod \"e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb\" (UID: \"7c584699-3a9f-45e1-a380-cb724ff74820\") " pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb"
Oct 30 19:37:36.216099 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.216008    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c584699-3a9f-45e1-a380-cb724ff74820-istio-podinfo\") pod \"e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb\" (UID: \"7c584699-3a9f-45e1-a380-cb724ff74820\") " pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb"
Oct 30 19:37:36.216099 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.216075    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c584699-3a9f-45e1-a380-cb724ff74820-istio-envoy\") pod \"e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb\" (UID: \"7c584699-3a9f-45e1-a380-cb724ff74820\") " pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb"
Oct 30 19:37:36.216806 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.216139    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c584699-3a9f-45e1-a380-cb724ff74820-workload-socket\") pod \"e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb\" (UID: \"7c584699-3a9f-45e1-a380-cb724ff74820\") " pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb"
Oct 30 19:37:36.216806 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.216187    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c584699-3a9f-45e1-a380-cb724ff74820-istio-data\") pod \"e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb\" (UID: \"7c584699-3a9f-45e1-a380-cb724ff74820\") " pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb"
Oct 30 19:37:36.216806 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.216259    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c584699-3a9f-45e1-a380-cb724ff74820-credential-socket\") pod \"e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb\" (UID: \"7c584699-3a9f-45e1-a380-cb724ff74820\") " pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb"
Oct 30 19:37:36.216806 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.216341    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c584699-3a9f-45e1-a380-cb724ff74820-workload-certs\") pod \"e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb\" (UID: \"7c584699-3a9f-45e1-a380-cb724ff74820\") " pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb"
Oct 30 19:37:36.216806 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.216367    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c584699-3a9f-45e1-a380-cb724ff74820-istiod-ca-cert\") pod \"e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb\" (UID: \"7c584699-3a9f-45e1-a380-cb724ff74820\") " pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb"
Oct 30 19:37:36.254807 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.217239    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c584699-3a9f-45e1-a380-cb724ff74820-istiod-ca-cert\") pod \"e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb\" (UID: \"7c584699-3a9f-45e1-a380-cb724ff74820\") " pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb"
Oct 30 19:37:36.254807 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.242336    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c584699-3a9f-45e1-a380-cb724ff74820-istio-data\") pod \"e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb\" (UID: \"7c584699-3a9f-45e1-a380-cb724ff74820\") " pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb"
Oct 30 19:37:36.254807 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.242978    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c584699-3a9f-45e1-a380-cb724ff74820-credential-socket\") pod \"e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb\" (UID: \"7c584699-3a9f-45e1-a380-cb724ff74820\") " pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb"
Oct 30 19:37:36.254807 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.243222    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c584699-3a9f-45e1-a380-cb724ff74820-workload-socket\") pod \"e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb\" (UID: \"7c584699-3a9f-45e1-a380-cb724ff74820\") " pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb"
Oct 30 19:37:36.254807 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.243433    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c584699-3a9f-45e1-a380-cb724ff74820-workload-certs\") pod \"e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb\" (UID: \"7c584699-3a9f-45e1-a380-cb724ff74820\") " pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb"
Oct 30 19:37:36.254807 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.247230    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c584699-3a9f-45e1-a380-cb724ff74820-istio-podinfo\") pod \"e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb\" (UID: \"7c584699-3a9f-45e1-a380-cb724ff74820\") " pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb"
Oct 30 19:37:36.261426 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.260674    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vw4c\" (UniqueName: \"kubernetes.io/projected/7c584699-3a9f-45e1-a380-cb724ff74820-kube-api-access-4vw4c\") pod \"e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb\" (UID: \"7c584699-3a9f-45e1-a380-cb724ff74820\") " pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb"
Oct 30 19:37:36.312813 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.312127    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c584699-3a9f-45e1-a380-cb724ff74820-istio-token\") pod \"e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb\" (UID: \"7c584699-3a9f-45e1-a380-cb724ff74820\") " pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb"
Oct 30 19:37:36.333763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.328036    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c584699-3a9f-45e1-a380-cb724ff74820-istio-envoy\") pod \"e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb\" (UID: \"7c584699-3a9f-45e1-a380-cb724ff74820\") " pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb"
Oct 30 19:37:36.354480 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.354108    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb"
Oct 30 19:37:36.444266 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.444227    5043 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Oct 30 19:37:36.444435 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.444290    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk" podUID="6ae382e3-be21-4e69-bf70-a8c0b8517789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Oct 30 19:37:36.659861 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.659829    5043 generic.go:356] "Generic (PLEG): container finished" podID="08851e5a-2c32-446b-9ffa-7508ac46cf2c" containerID="a5f721e18202d159d86aa018d44573d4d42069066d22449b917f5087fae5ebf7" exitCode=0
Oct 30 19:37:36.662867 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.662842    5043 generic.go:356] "Generic (PLEG): container finished" podID="87ce476f-991e-4437-a15e-f7410d2392a5" containerID="95ca287b5e5a4666f51105382ef510973597c87b3d018c47352ad5901dd830d3" exitCode=0
Oct 30 19:37:36.670014 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.669992    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="224648b8-553f-44ad-9d9a-3784a7bea898" path="/var/lib/kubelet/pods/224648b8-553f-44ad-9d9a-3784a7bea898/volumes"
Oct 30 19:37:36.671007 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.670987    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="359c47fc-f8af-43f8-865b-1f187fb2672c" path="/var/lib/kubelet/pods/359c47fc-f8af-43f8-865b-1f187fb2672c/volumes"
Oct 30 19:37:36.671835 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.671817    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fea9e2a-77c0-48c4-8517-3e104fc043b8" path="/var/lib/kubelet/pods/4fea9e2a-77c0-48c4-8517-3e104fc043b8/volumes"
Oct 30 19:37:36.673132 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.673114    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ce15b3c-1548-489c-a4f1-2fc151a53d35" path="/var/lib/kubelet/pods/8ce15b3c-1548-489c-a4f1-2fc151a53d35/volumes"
Oct 30 19:37:36.673879 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.673862    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a851569d-8aa4-4af2-9f3c-c90c4895b95a" path="/var/lib/kubelet/pods/a851569d-8aa4-4af2-9f3c-c90c4895b95a/volumes"
Oct 30 19:37:36.674722 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.674704    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3185a2a-d994-41d4-a8d7-9b7e966b9f1c" path="/var/lib/kubelet/pods/c3185a2a-d994-41d4-a8d7-9b7e966b9f1c/volumes"
Oct 30 19:37:36.675964 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.675944    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8a7200d-6acf-4008-8156-d9102730ac76" path="/var/lib/kubelet/pods/d8a7200d-6acf-4008-8156-d9102730ac76/volumes"
Oct 30 19:37:36.676549 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.676524    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-194e54e4-9dd4p" event={"ID":"08851e5a-2c32-446b-9ffa-7508ac46cf2c","Type":"ContainerDied","Data":"a5f721e18202d159d86aa018d44573d4d42069066d22449b917f5087fae5ebf7"}
Oct 30 19:37:36.676713 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.676694    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6266b58-dd5hf" event={"ID":"87ce476f-991e-4437-a15e-f7410d2392a5","Type":"ContainerDied","Data":"95ca287b5e5a4666f51105382ef510973597c87b3d018c47352ad5901dd830d3"}
Oct 30 19:37:36.915007 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.911229    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb"]
Oct 30 19:37:36.930868 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.929825    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:37:36.930868 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.929931    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:37:36.930868 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:36.929977    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:37:37.442175 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:37.442143    5043 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Oct 30 19:37:37.442582 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:37.442196    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk" podUID="6ae382e3-be21-4e69-bf70-a8c0b8517789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Oct 30 19:37:37.717389 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:37.717355    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb" event={"ID":"7c584699-3a9f-45e1-a380-cb724ff74820","Type":"ContainerStarted","Data":"0ca07795e91bd72eaa34df2f3bb17a19d451880cb6df0d83257b6302915182b7"}
Oct 30 19:37:37.717553 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:37.717426    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb" event={"ID":"7c584699-3a9f-45e1-a380-cb724ff74820","Type":"ContainerStarted","Data":"e88470840cb59d88b988de45e15f4233527e1621b82cd4b53bc2e1f57af24e5b"}
Oct 30 19:37:37.802311 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:37.802234    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb" podStartSLOduration=2.802198002 podStartE2EDuration="2.802198002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:37.797422095 +0000 UTC m=+1277.308482060" watchObservedRunningTime="2026-10-30 19:37:37.802198002 +0000 UTC m=+1277.313257968"
Oct 30 19:37:37.944109 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:37.943262    5043 patch_prober.go:29] interesting pod/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:37:37.944109 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:37.943544    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk" podUID="c08cde99-b619-487f-b34f-3180ba5019be" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:37:38.162976 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:38.162941    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-194e54e4-9dd4p"
Oct 30 19:37:38.235827 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:38.235451    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6266b58-dd5hf"
Oct 30 19:37:38.241448 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:38.241416    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08851e5a-2c32-446b-9ffa-7508ac46cf2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08851e5a-2c32-446b-9ffa-7508ac46cf2c-maas-api-openshift-ca\") pod \"08851e5a-2c32-446b-9ffa-7508ac46cf2c\" (UID: \"08851e5a-2c32-446b-9ffa-7508ac46cf2c\") "
Oct 30 19:37:38.242434 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:38.242408    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08851e5a-2c32-446b-9ffa-7508ac46cf2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08851e5a-2c32-446b-9ffa-7508ac46cf2c-maas-api-xks-ca\") pod \"08851e5a-2c32-446b-9ffa-7508ac46cf2c\" (UID: \"08851e5a-2c32-446b-9ffa-7508ac46cf2c\") "
Oct 30 19:37:38.243004 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:38.241993    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08851e5a-2c32-446b-9ffa-7508ac46cf2c-maas-api-openshift-ca" pod "08851e5a-2c32-446b-9ffa-7508ac46cf2c" (UID: "08851e5a-2c32-446b-9ffa-7508ac46cf2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:37:38.243219 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:38.243201    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08851e5a-2c32-446b-9ffa-7508ac46cf2c-kube-api-access-5jcl7\" (UniqueName: \"kubernetes.io/projected/08851e5a-2c32-446b-9ffa-7508ac46cf2c-kube-api-access-5jcl7\") pod \"08851e5a-2c32-446b-9ffa-7508ac46cf2c\" (UID: \"08851e5a-2c32-446b-9ffa-7508ac46cf2c\") "
Oct 30 19:37:38.244508 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:38.244486    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08851e5a-2c32-446b-9ffa-7508ac46cf2c-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:38.245431 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:38.245408    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08851e5a-2c32-446b-9ffa-7508ac46cf2c-maas-api-xks-ca" pod "08851e5a-2c32-446b-9ffa-7508ac46cf2c" (UID: "08851e5a-2c32-446b-9ffa-7508ac46cf2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:37:38.246098 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:38.246064    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08851e5a-2c32-446b-9ffa-7508ac46cf2c-kube-api-access-5jcl7" pod "08851e5a-2c32-446b-9ffa-7508ac46cf2c" (UID: "08851e5a-2c32-446b-9ffa-7508ac46cf2c"). InnerVolumeSpecName "kube-api-access-5jcl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:38.347128 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:38.346189    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87ce476f-991e-4437-a15e-f7410d2392a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87ce476f-991e-4437-a15e-f7410d2392a5-maas-api-xks-ca\") pod \"87ce476f-991e-4437-a15e-f7410d2392a5\" (UID: \"87ce476f-991e-4437-a15e-f7410d2392a5\") "
Oct 30 19:37:38.347128 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:38.346515    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87ce476f-991e-4437-a15e-f7410d2392a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87ce476f-991e-4437-a15e-f7410d2392a5-maas-api-openshift-ca\") pod \"87ce476f-991e-4437-a15e-f7410d2392a5\" (UID: \"87ce476f-991e-4437-a15e-f7410d2392a5\") "
Oct 30 19:37:38.347128 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:38.346634    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87ce476f-991e-4437-a15e-f7410d2392a5-kube-api-access-m8rjn\" (UniqueName: \"kubernetes.io/projected/87ce476f-991e-4437-a15e-f7410d2392a5-kube-api-access-m8rjn\") pod \"87ce476f-991e-4437-a15e-f7410d2392a5\" (UID: \"87ce476f-991e-4437-a15e-f7410d2392a5\") "
Oct 30 19:37:38.347794 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:38.347432    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87ce476f-991e-4437-a15e-f7410d2392a5-maas-api-openshift-ca" pod "87ce476f-991e-4437-a15e-f7410d2392a5" (UID: "87ce476f-991e-4437-a15e-f7410d2392a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:37:38.348406 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:38.348320    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87ce476f-991e-4437-a15e-f7410d2392a5-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:38.348406 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:38.348357    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08851e5a-2c32-446b-9ffa-7508ac46cf2c-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:38.348406 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:38.348374    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jcl7\" (UniqueName: \"kubernetes.io/projected/08851e5a-2c32-446b-9ffa-7508ac46cf2c-kube-api-access-5jcl7\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:38.350157 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:38.350127    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87ce476f-991e-4437-a15e-f7410d2392a5-maas-api-xks-ca" pod "87ce476f-991e-4437-a15e-f7410d2392a5" (UID: "87ce476f-991e-4437-a15e-f7410d2392a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:37:38.351007 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:38.350973    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87ce476f-991e-4437-a15e-f7410d2392a5-kube-api-access-m8rjn" pod "87ce476f-991e-4437-a15e-f7410d2392a5" (UID: "87ce476f-991e-4437-a15e-f7410d2392a5"). InnerVolumeSpecName "kube-api-access-m8rjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:38.355287 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:38.355260    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb"
Oct 30 19:37:38.386893 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:38.386108    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb"
Oct 30 19:37:38.442153 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:38.442117    5043 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Oct 30 19:37:38.442337 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:38.442182    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk" podUID="6ae382e3-be21-4e69-bf70-a8c0b8517789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Oct 30 19:37:38.455027 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:38.454140    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8rjn\" (UniqueName: \"kubernetes.io/projected/87ce476f-991e-4437-a15e-f7410d2392a5-kube-api-access-m8rjn\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:38.455027 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:38.454183    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87ce476f-991e-4437-a15e-f7410d2392a5-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:38.739434 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:38.739402    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-194e54e4-9dd4p"
Oct 30 19:37:38.739613 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:38.739461    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-194e54e4-9dd4p" event={"ID":"08851e5a-2c32-446b-9ffa-7508ac46cf2c","Type":"ContainerDied","Data":"1e133cbd305c8246738c23f6d01278ae191fd786c5920f4691c59b872d0afa72"}
Oct 30 19:37:38.739613 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:38.739521    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e133cbd305c8246738c23f6d01278ae191fd786c5920f4691c59b872d0afa72"
Oct 30 19:37:38.741285 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:38.741257    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6266b58-dd5hf"
Oct 30 19:37:38.741447 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:38.741292    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6266b58-dd5hf" event={"ID":"87ce476f-991e-4437-a15e-f7410d2392a5","Type":"ContainerDied","Data":"8ca7c9b0b2a8f7cfa2c9e3beae99541b7ff1f4c738666327e7a1641a01789b75"}
Oct 30 19:37:38.741447 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:38.741435    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ca7c9b0b2a8f7cfa2c9e3beae99541b7ff1f4c738666327e7a1641a01789b75"
Oct 30 19:37:38.742026 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:38.742005    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb"
Oct 30 19:37:38.743134 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:38.743114    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb"
Oct 30 19:37:39.446657 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:39.446626    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk"
Oct 30 19:37:39.447142 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:39.447075    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk"
Oct 30 19:37:39.447907 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:39.447886    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk"
Oct 30 19:37:40.262122 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:40.262096    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk"
Oct 30 19:37:40.406896 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:40.406865    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c08cde99-b619-487f-b34f-3180ba5019be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c08cde99-b619-487f-b34f-3180ba5019be-istio-envoy\") pod \"c08cde99-b619-487f-b34f-3180ba5019be\" (UID: \"c08cde99-b619-487f-b34f-3180ba5019be\") "
Oct 30 19:37:40.407085 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:40.406998    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c08cde99-b619-487f-b34f-3180ba5019be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c08cde99-b619-487f-b34f-3180ba5019be-istiod-ca-cert\") pod \"c08cde99-b619-487f-b34f-3180ba5019be\" (UID: \"c08cde99-b619-487f-b34f-3180ba5019be\") "
Oct 30 19:37:40.407133 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:40.407087    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c08cde99-b619-487f-b34f-3180ba5019be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08cde99-b619-487f-b34f-3180ba5019be-credential-socket\") pod \"c08cde99-b619-487f-b34f-3180ba5019be\" (UID: \"c08cde99-b619-487f-b34f-3180ba5019be\") "
Oct 30 19:37:40.407324 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:40.407303    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c08cde99-b619-487f-b34f-3180ba5019be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c08cde99-b619-487f-b34f-3180ba5019be-istio-podinfo\") pod \"c08cde99-b619-487f-b34f-3180ba5019be\" (UID: \"c08cde99-b619-487f-b34f-3180ba5019be\") "
Oct 30 19:37:40.407397 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:40.407362    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c08cde99-b619-487f-b34f-3180ba5019be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08cde99-b619-487f-b34f-3180ba5019be-workload-socket\") pod \"c08cde99-b619-487f-b34f-3180ba5019be\" (UID: \"c08cde99-b619-487f-b34f-3180ba5019be\") "
Oct 30 19:37:40.407397 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:40.407357    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08cde99-b619-487f-b34f-3180ba5019be-credential-socket" pod "c08cde99-b619-487f-b34f-3180ba5019be" (UID: "c08cde99-b619-487f-b34f-3180ba5019be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:40.407397 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:40.407380    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c08cde99-b619-487f-b34f-3180ba5019be-istiod-ca-cert" pod "c08cde99-b619-487f-b34f-3180ba5019be" (UID: "c08cde99-b619-487f-b34f-3180ba5019be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:37:40.407561 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:40.407383    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c08cde99-b619-487f-b34f-3180ba5019be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c08cde99-b619-487f-b34f-3180ba5019be-workload-certs\") pod \"c08cde99-b619-487f-b34f-3180ba5019be\" (UID: \"c08cde99-b619-487f-b34f-3180ba5019be\") "
Oct 30 19:37:40.407561 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:40.407443    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c08cde99-b619-487f-b34f-3180ba5019be-kube-api-access-7swqx\" (UniqueName: \"kubernetes.io/projected/c08cde99-b619-487f-b34f-3180ba5019be-kube-api-access-7swqx\") pod \"c08cde99-b619-487f-b34f-3180ba5019be\" (UID: \"c08cde99-b619-487f-b34f-3180ba5019be\") "
Oct 30 19:37:40.407670 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:40.407482    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c08cde99-b619-487f-b34f-3180ba5019be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c08cde99-b619-487f-b34f-3180ba5019be-istio-data\") pod \"c08cde99-b619-487f-b34f-3180ba5019be\" (UID: \"c08cde99-b619-487f-b34f-3180ba5019be\") "
Oct 30 19:37:40.407670 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:40.407598    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c08cde99-b619-487f-b34f-3180ba5019be-istio-token\" (UniqueName: \"kubernetes.io/projected/c08cde99-b619-487f-b34f-3180ba5019be-istio-token\") pod \"c08cde99-b619-487f-b34f-3180ba5019be\" (UID: \"c08cde99-b619-487f-b34f-3180ba5019be\") "
Oct 30 19:37:40.407670 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:40.407549    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08cde99-b619-487f-b34f-3180ba5019be-workload-certs" pod "c08cde99-b619-487f-b34f-3180ba5019be" (UID: "c08cde99-b619-487f-b34f-3180ba5019be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:40.407670 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:40.407598    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08cde99-b619-487f-b34f-3180ba5019be-workload-socket" pod "c08cde99-b619-487f-b34f-3180ba5019be" (UID: "c08cde99-b619-487f-b34f-3180ba5019be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:40.407991 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:40.407696    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08cde99-b619-487f-b34f-3180ba5019be-istio-data" pod "c08cde99-b619-487f-b34f-3180ba5019be" (UID: "c08cde99-b619-487f-b34f-3180ba5019be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:40.408325 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:40.408299    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08cde99-b619-487f-b34f-3180ba5019be-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:40.408448 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:40.408330    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c08cde99-b619-487f-b34f-3180ba5019be-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:40.408448 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:40.408347    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c08cde99-b619-487f-b34f-3180ba5019be-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:40.408448 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:40.408360    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c08cde99-b619-487f-b34f-3180ba5019be-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:40.408448 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:40.408375    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08cde99-b619-487f-b34f-3180ba5019be-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:40.409955 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:40.409934    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08cde99-b619-487f-b34f-3180ba5019be-istio-envoy" pod "c08cde99-b619-487f-b34f-3180ba5019be" (UID: "c08cde99-b619-487f-b34f-3180ba5019be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:40.410090 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:40.410065    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c08cde99-b619-487f-b34f-3180ba5019be-istio-token" pod "c08cde99-b619-487f-b34f-3180ba5019be" (UID: "c08cde99-b619-487f-b34f-3180ba5019be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:40.410453 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:40.410430    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c08cde99-b619-487f-b34f-3180ba5019be-istio-podinfo" pod "c08cde99-b619-487f-b34f-3180ba5019be" (UID: "c08cde99-b619-487f-b34f-3180ba5019be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:37:40.410520 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:40.410460    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c08cde99-b619-487f-b34f-3180ba5019be-kube-api-access-7swqx" pod "c08cde99-b619-487f-b34f-3180ba5019be" (UID: "c08cde99-b619-487f-b34f-3180ba5019be"). InnerVolumeSpecName "kube-api-access-7swqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:40.509791 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:40.509672    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c08cde99-b619-487f-b34f-3180ba5019be-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:40.509791 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:40.509707    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7swqx\" (UniqueName: \"kubernetes.io/projected/c08cde99-b619-487f-b34f-3180ba5019be-kube-api-access-7swqx\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:40.509791 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:40.509725    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c08cde99-b619-487f-b34f-3180ba5019be-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:40.509791 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:40.509757    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c08cde99-b619-487f-b34f-3180ba5019be-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:40.890819 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:40.890099    5043 generic.go:356] "Generic (PLEG): container finished" podID="c08cde99-b619-487f-b34f-3180ba5019be" containerID="e1fa23bc7e10f805bb6e9542b2743e6d7381d059afc293bd5967426fdbcc01d8" exitCode=0
Oct 30 19:37:40.906242 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:40.891221    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk"
Oct 30 19:37:40.911861 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:40.891487    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk" event={"ID":"c08cde99-b619-487f-b34f-3180ba5019be","Type":"ContainerDied","Data":"e1fa23bc7e10f805bb6e9542b2743e6d7381d059afc293bd5967426fdbcc01d8"}
Oct 30 19:37:40.911861 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:40.909298    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk" event={"ID":"c08cde99-b619-487f-b34f-3180ba5019be","Type":"ContainerDied","Data":"e7d27ede1afb947ab260196cc8c5b176bf120fe76b394342f56a314a055518bb"}
Oct 30 19:37:40.911861 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:40.909361    5043 scope.go:122] "RemoveContainer" containerID="e1fa23bc7e10f805bb6e9542b2743e6d7381d059afc293bd5967426fdbcc01d8"
Oct 30 19:37:40.930852 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:40.930828    5043 scope.go:122] "RemoveContainer" containerID="a21e899d404b92394a64820357f8a6e4b28ee883343ff5bd273ae1759465914b"
Oct 30 19:37:40.942837 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:40.942796    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk"]
Oct 30 19:37:40.955970 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:40.955927    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-95d2ac-openshift-default-7f6cb95d55-t6xhk"]
Oct 30 19:37:40.977207 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:40.977164    5043 scope.go:122] "RemoveContainer" containerID="e1fa23bc7e10f805bb6e9542b2743e6d7381d059afc293bd5967426fdbcc01d8"
Oct 30 19:37:40.977826 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:37:40.977794    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1fa23bc7e10f805bb6e9542b2743e6d7381d059afc293bd5967426fdbcc01d8\": container with ID starting with e1fa23bc7e10f805bb6e9542b2743e6d7381d059afc293bd5967426fdbcc01d8 not found: ID does not exist" containerID="e1fa23bc7e10f805bb6e9542b2743e6d7381d059afc293bd5967426fdbcc01d8"
Oct 30 19:37:40.977923 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:40.977867    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1fa23bc7e10f805bb6e9542b2743e6d7381d059afc293bd5967426fdbcc01d8"} err="failed to get container status \"e1fa23bc7e10f805bb6e9542b2743e6d7381d059afc293bd5967426fdbcc01d8\": rpc error: code = NotFound desc = could not find container \"e1fa23bc7e10f805bb6e9542b2743e6d7381d059afc293bd5967426fdbcc01d8\": container with ID starting with e1fa23bc7e10f805bb6e9542b2743e6d7381d059afc293bd5967426fdbcc01d8 not found: ID does not exist"
Oct 30 19:37:40.978019 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:40.977935    5043 scope.go:122] "RemoveContainer" containerID="a21e899d404b92394a64820357f8a6e4b28ee883343ff5bd273ae1759465914b"
Oct 30 19:37:40.978721 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:37:40.978690    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a21e899d404b92394a64820357f8a6e4b28ee883343ff5bd273ae1759465914b\": container with ID starting with a21e899d404b92394a64820357f8a6e4b28ee883343ff5bd273ae1759465914b not found: ID does not exist" containerID="a21e899d404b92394a64820357f8a6e4b28ee883343ff5bd273ae1759465914b"
Oct 30 19:37:40.978823 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:40.978747    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a21e899d404b92394a64820357f8a6e4b28ee883343ff5bd273ae1759465914b"} err="failed to get container status \"a21e899d404b92394a64820357f8a6e4b28ee883343ff5bd273ae1759465914b\": rpc error: code = NotFound desc = could not find container \"a21e899d404b92394a64820357f8a6e4b28ee883343ff5bd273ae1759465914b\": container with ID starting with a21e899d404b92394a64820357f8a6e4b28ee883343ff5bd273ae1759465914b not found: ID does not exist"
Oct 30 19:37:41.675396 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:41.675361    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-579ddb-9b6756cf9-gfdfg"]
Oct 30 19:37:41.676571 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:41.676543    5043 cpu_manager.go:414] "containerMap: removing container" podUID="87ce476f-991e-4437-a15e-f7410d2392a5" containerName="revoke-keys"
Oct 30 19:37:41.676571 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:41.676567    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87ce476f-991e-4437-a15e-f7410d2392a5" containerName="revoke-keys"
Oct 30 19:37:41.676717 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:41.676613    5043 cpu_manager.go:414] "containerMap: removing container" podUID="08851e5a-2c32-446b-9ffa-7508ac46cf2c" containerName="revoke-keys"
Oct 30 19:37:41.676717 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:41.676621    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08851e5a-2c32-446b-9ffa-7508ac46cf2c" containerName="revoke-keys"
Oct 30 19:37:41.676717 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:41.676655    5043 cpu_manager.go:414] "containerMap: removing container" podUID="c08cde99-b619-487f-b34f-3180ba5019be" containerName="istio-proxy"
Oct 30 19:37:41.676717 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:41.676664    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c08cde99-b619-487f-b34f-3180ba5019be" containerName="istio-proxy"
Oct 30 19:37:41.676717 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:41.676690    5043 cpu_manager.go:414] "containerMap: removing container" podUID="c08cde99-b619-487f-b34f-3180ba5019be" containerName="istio-proxy"
Oct 30 19:37:41.676717 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:41.676696    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c08cde99-b619-487f-b34f-3180ba5019be" containerName="istio-proxy"
Oct 30 19:37:41.677047 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:41.676874    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="87ce476f-991e-4437-a15e-f7410d2392a5" containerName="revoke-keys"
Oct 30 19:37:41.677047 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:41.676892    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="08851e5a-2c32-446b-9ffa-7508ac46cf2c" containerName="revoke-keys"
Oct 30 19:37:41.677047 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:41.676905    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="224648b8-553f-44ad-9d9a-3784a7bea898" containerName="istio-proxy"
Oct 30 19:37:41.677047 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:41.676918    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="c08cde99-b619-487f-b34f-3180ba5019be" containerName="istio-proxy"
Oct 30 19:37:41.677047 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:41.676930    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="c08cde99-b619-487f-b34f-3180ba5019be" containerName="istio-proxy"
Oct 30 19:37:41.683309 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:41.683284    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-579ddb-9b6756cf9-gfdfg"
Oct 30 19:37:41.689962 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:41.689930    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-579ddb-9b6756cf9-gfdfg"]
Oct 30 19:37:41.695172 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:41.693872    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-579ddb\""
Oct 30 19:37:41.728036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:41.728003    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-579ddb-6756775486-ghzlj"]
Oct 30 19:37:41.734142 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:41.734115    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-579ddb-6756775486-ghzlj"
Oct 30 19:37:41.752650 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:41.751497    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-579ddb-6756775486-ghzlj"]
Oct 30 19:37:41.831772 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:41.823220    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-579ddb-dockercfg-684sr\""
Oct 30 19:37:41.845403 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:41.843932    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38d6d4ee-f2f5-4747-9e4d-c371701193ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-579ddb-9b6756cf9-gfdfg\" (UID: \"38d6d4ee-f2f5-4747-9e4d-c371701193ad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-579ddb-9b6756cf9-gfdfg"
Oct 30 19:37:41.845403 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:41.844147    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4wdm\" (UniqueName: \"kubernetes.io/projected/8861a11f-5436-41be-9c85-15a114f3b4cd-kube-api-access-x4wdm\") pod \"payload-processing-e2e-inf-a-579ddb-6756775486-ghzlj\" (UID: \"8861a11f-5436-41be-9c85-15a114f3b4cd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-579ddb-6756775486-ghzlj"
Oct 30 19:37:41.845403 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:41.844295    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d595\" (UniqueName: \"kubernetes.io/projected/38d6d4ee-f2f5-4747-9e4d-c371701193ad-kube-api-access-4d595\") pod \"payload-pre-processing-e2e-inf-a-579ddb-9b6756cf9-gfdfg\" (UID: \"38d6d4ee-f2f5-4747-9e4d-c371701193ad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-579ddb-9b6756cf9-gfdfg"
Oct 30 19:37:41.845403 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:41.844409    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8861a11f-5436-41be-9c85-15a114f3b4cd-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-579ddb-6756775486-ghzlj\" (UID: \"8861a11f-5436-41be-9c85-15a114f3b4cd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-579ddb-6756775486-ghzlj"
Oct 30 19:37:41.993890 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:41.978796    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d595\" (UniqueName: \"kubernetes.io/projected/38d6d4ee-f2f5-4747-9e4d-c371701193ad-kube-api-access-4d595\") pod \"payload-pre-processing-e2e-inf-a-579ddb-9b6756cf9-gfdfg\" (UID: \"38d6d4ee-f2f5-4747-9e4d-c371701193ad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-579ddb-9b6756cf9-gfdfg"
Oct 30 19:37:41.993890 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:41.980968    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8861a11f-5436-41be-9c85-15a114f3b4cd-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-579ddb-6756775486-ghzlj\" (UID: \"8861a11f-5436-41be-9c85-15a114f3b4cd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-579ddb-6756775486-ghzlj"
Oct 30 19:37:41.993890 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:41.981088    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38d6d4ee-f2f5-4747-9e4d-c371701193ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-579ddb-9b6756cf9-gfdfg\" (UID: \"38d6d4ee-f2f5-4747-9e4d-c371701193ad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-579ddb-9b6756cf9-gfdfg"
Oct 30 19:37:41.993890 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:41.981274    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4wdm\" (UniqueName: \"kubernetes.io/projected/8861a11f-5436-41be-9c85-15a114f3b4cd-kube-api-access-x4wdm\") pod \"payload-processing-e2e-inf-a-579ddb-6756775486-ghzlj\" (UID: \"8861a11f-5436-41be-9c85-15a114f3b4cd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-579ddb-6756775486-ghzlj"
Oct 30 19:37:41.993890 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:41.982476    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8861a11f-5436-41be-9c85-15a114f3b4cd-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-579ddb-6756775486-ghzlj\" (UID: \"8861a11f-5436-41be-9c85-15a114f3b4cd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-579ddb-6756775486-ghzlj"
Oct 30 19:37:41.993890 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:41.983493    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38d6d4ee-f2f5-4747-9e4d-c371701193ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-579ddb-9b6756cf9-gfdfg\" (UID: \"38d6d4ee-f2f5-4747-9e4d-c371701193ad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-579ddb-9b6756cf9-gfdfg"
Oct 30 19:37:42.108861 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:42.102495    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d595\" (UniqueName: \"kubernetes.io/projected/38d6d4ee-f2f5-4747-9e4d-c371701193ad-kube-api-access-4d595\") pod \"payload-pre-processing-e2e-inf-a-579ddb-9b6756cf9-gfdfg\" (UID: \"38d6d4ee-f2f5-4747-9e4d-c371701193ad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-579ddb-9b6756cf9-gfdfg"
Oct 30 19:37:42.108861 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:42.106305    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4wdm\" (UniqueName: \"kubernetes.io/projected/8861a11f-5436-41be-9c85-15a114f3b4cd-kube-api-access-x4wdm\") pod \"payload-processing-e2e-inf-a-579ddb-6756775486-ghzlj\" (UID: \"8861a11f-5436-41be-9c85-15a114f3b4cd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-579ddb-6756775486-ghzlj"
Oct 30 19:37:42.186474 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:42.185314    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-579ddb-6756775486-ghzlj"
Oct 30 19:37:42.311728 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:42.311394    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-579ddb-9b6756cf9-gfdfg"
Oct 30 19:37:42.570385 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:37:42.570349    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8861a11f_5436_41be_9c85_15a114f3b4cd.slice/crio-aacf17b9dbc65bfe283cfd33a5d8196eef8b246d2ad79706c4e076a293abfce8 WatchSource:0}: Error finding container aacf17b9dbc65bfe283cfd33a5d8196eef8b246d2ad79706c4e076a293abfce8: Status 404 returned error can't find the container with id aacf17b9dbc65bfe283cfd33a5d8196eef8b246d2ad79706c4e076a293abfce8
Oct 30 19:37:42.577881 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:42.576256    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-579ddb-6756775486-ghzlj"]
Oct 30 19:37:42.650581 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:42.650545    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c08cde99-b619-487f-b34f-3180ba5019be" path="/var/lib/kubelet/pods/c08cde99-b619-487f-b34f-3180ba5019be/volumes"
Oct 30 19:37:42.656295 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:42.656255    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-579ddb-9b6756cf9-gfdfg"]
Oct 30 19:37:42.657945 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:37:42.657910    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38d6d4ee_f2f5_4747_9e4d_c371701193ad.slice/crio-7e8aab8797664798d7fb89242917c866b8498bddca8008e7c0601bfd52127852 WatchSource:0}: Error finding container 7e8aab8797664798d7fb89242917c866b8498bddca8008e7c0601bfd52127852: Status 404 returned error can't find the container with id 7e8aab8797664798d7fb89242917c866b8498bddca8008e7c0601bfd52127852
Oct 30 19:37:43.032440 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:43.032404    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-579ddb-9b6756cf9-gfdfg" event={"ID":"38d6d4ee-f2f5-4747-9e4d-c371701193ad","Type":"ContainerStarted","Data":"b63836170c1831649ad5c2f5df15a39a5a3788651b7f46138a2e545cc1c86746"}
Oct 30 19:37:43.032922 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:43.032472    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-579ddb-9b6756cf9-gfdfg" event={"ID":"38d6d4ee-f2f5-4747-9e4d-c371701193ad","Type":"ContainerStarted","Data":"7e8aab8797664798d7fb89242917c866b8498bddca8008e7c0601bfd52127852"}
Oct 30 19:37:43.032922 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:43.032517    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-579ddb-9b6756cf9-gfdfg"
Oct 30 19:37:43.035642 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:43.035614    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-579ddb-6756775486-ghzlj" event={"ID":"8861a11f-5436-41be-9c85-15a114f3b4cd","Type":"ContainerStarted","Data":"912039d27946c685339eeb37352ba2124eff68cb08df6b9564323935cbe8d857"}
Oct 30 19:37:43.035784 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:43.035666    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-579ddb-6756775486-ghzlj" event={"ID":"8861a11f-5436-41be-9c85-15a114f3b4cd","Type":"ContainerStarted","Data":"aacf17b9dbc65bfe283cfd33a5d8196eef8b246d2ad79706c4e076a293abfce8"}
Oct 30 19:37:43.035784 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:43.035701    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-579ddb-6756775486-ghzlj"
Oct 30 19:37:43.051015 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:43.050965    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-579ddb-9b6756cf9-gfdfg" podStartSLOduration=2.050939199 podStartE2EDuration="2.050939199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:43.048757183 +0000 UTC m=+1282.559817142" watchObservedRunningTime="2026-10-30 19:37:43.050939199 +0000 UTC m=+1282.561999163"
Oct 30 19:37:43.073515 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:43.073463    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-579ddb-6756775486-ghzlj" podStartSLOduration=2.073448282 podStartE2EDuration="2.073448282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:43.067830644 +0000 UTC m=+1282.578890609" watchObservedRunningTime="2026-10-30 19:37:43.073448282 +0000 UTC m=+1282.584508247"
Oct 30 19:37:43.215285 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:43.215252    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-73f96d-szmhm"]
Oct 30 19:37:43.219576 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:43.219552    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-73f96d-szmhm"]
Oct 30 19:37:44.648960 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:44.648924    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a7c6064-437f-46b9-93bd-a9b2cb31ede3" path="/var/lib/kubelet/pods/1a7c6064-437f-46b9-93bd-a9b2cb31ede3/volumes"
Oct 30 19:37:46.145500 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:46.145464    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9"]
Oct 30 19:37:46.151326 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:46.151297    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9"
Oct 30 19:37:46.175895 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:46.175863    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9"]
Oct 30 19:37:46.304256 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:46.304214    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8f2440f-1238-46f6-ba78-90f9c38b574d-istio-data\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9\" (UID: \"e8f2440f-1238-46f6-ba78-90f9c38b574d\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9"
Oct 30 19:37:46.304473 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:46.304445    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8f2440f-1238-46f6-ba78-90f9c38b574d-istio-podinfo\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9\" (UID: \"e8f2440f-1238-46f6-ba78-90f9c38b574d\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9"
Oct 30 19:37:46.304559 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:46.304541    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8f2440f-1238-46f6-ba78-90f9c38b574d-istio-envoy\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9\" (UID: \"e8f2440f-1238-46f6-ba78-90f9c38b574d\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9"
Oct 30 19:37:46.304625 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:46.304613    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8f2440f-1238-46f6-ba78-90f9c38b574d-istio-token\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9\" (UID: \"e8f2440f-1238-46f6-ba78-90f9c38b574d\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9"
Oct 30 19:37:46.304769 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:46.304726    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8f2440f-1238-46f6-ba78-90f9c38b574d-workload-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9\" (UID: \"e8f2440f-1238-46f6-ba78-90f9c38b574d\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9"
Oct 30 19:37:46.304924 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:46.304799    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8f2440f-1238-46f6-ba78-90f9c38b574d-istiod-ca-cert\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9\" (UID: \"e8f2440f-1238-46f6-ba78-90f9c38b574d\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9"
Oct 30 19:37:46.304924 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:46.304846    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6w8l\" (UniqueName: \"kubernetes.io/projected/e8f2440f-1238-46f6-ba78-90f9c38b574d-kube-api-access-n6w8l\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9\" (UID: \"e8f2440f-1238-46f6-ba78-90f9c38b574d\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9"
Oct 30 19:37:46.305047 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:46.304922    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8f2440f-1238-46f6-ba78-90f9c38b574d-workload-certs\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9\" (UID: \"e8f2440f-1238-46f6-ba78-90f9c38b574d\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9"
Oct 30 19:37:46.305047 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:46.304943    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8f2440f-1238-46f6-ba78-90f9c38b574d-credential-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9\" (UID: \"e8f2440f-1238-46f6-ba78-90f9c38b574d\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9"
Oct 30 19:37:46.406995 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:46.406881    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8f2440f-1238-46f6-ba78-90f9c38b574d-istio-podinfo\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9\" (UID: \"e8f2440f-1238-46f6-ba78-90f9c38b574d\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9"
Oct 30 19:37:46.406995 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:46.406966    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8f2440f-1238-46f6-ba78-90f9c38b574d-istio-envoy\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9\" (UID: \"e8f2440f-1238-46f6-ba78-90f9c38b574d\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9"
Oct 30 19:37:46.407245 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:46.407032    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8f2440f-1238-46f6-ba78-90f9c38b574d-istio-token\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9\" (UID: \"e8f2440f-1238-46f6-ba78-90f9c38b574d\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9"
Oct 30 19:37:46.407245 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:46.407151    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8f2440f-1238-46f6-ba78-90f9c38b574d-workload-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9\" (UID: \"e8f2440f-1238-46f6-ba78-90f9c38b574d\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9"
Oct 30 19:37:46.407245 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:46.407206    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8f2440f-1238-46f6-ba78-90f9c38b574d-istiod-ca-cert\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9\" (UID: \"e8f2440f-1238-46f6-ba78-90f9c38b574d\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9"
Oct 30 19:37:46.407404 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:46.407275    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6w8l\" (UniqueName: \"kubernetes.io/projected/e8f2440f-1238-46f6-ba78-90f9c38b574d-kube-api-access-n6w8l\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9\" (UID: \"e8f2440f-1238-46f6-ba78-90f9c38b574d\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9"
Oct 30 19:37:46.407404 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:46.407307    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8f2440f-1238-46f6-ba78-90f9c38b574d-workload-certs\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9\" (UID: \"e8f2440f-1238-46f6-ba78-90f9c38b574d\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9"
Oct 30 19:37:46.407404 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:46.407337    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8f2440f-1238-46f6-ba78-90f9c38b574d-credential-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9\" (UID: \"e8f2440f-1238-46f6-ba78-90f9c38b574d\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9"
Oct 30 19:37:46.408161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:46.408134    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8f2440f-1238-46f6-ba78-90f9c38b574d-istiod-ca-cert\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9\" (UID: \"e8f2440f-1238-46f6-ba78-90f9c38b574d\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9"
Oct 30 19:37:46.408326 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:46.408303    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8f2440f-1238-46f6-ba78-90f9c38b574d-istio-data\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9\" (UID: \"e8f2440f-1238-46f6-ba78-90f9c38b574d\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9"
Oct 30 19:37:46.408921 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:46.408544    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8f2440f-1238-46f6-ba78-90f9c38b574d-workload-certs\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9\" (UID: \"e8f2440f-1238-46f6-ba78-90f9c38b574d\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9"
Oct 30 19:37:46.408921 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:46.408703    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8f2440f-1238-46f6-ba78-90f9c38b574d-istio-data\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9\" (UID: \"e8f2440f-1238-46f6-ba78-90f9c38b574d\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9"
Oct 30 19:37:46.408921 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:46.408855    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8f2440f-1238-46f6-ba78-90f9c38b574d-credential-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9\" (UID: \"e8f2440f-1238-46f6-ba78-90f9c38b574d\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9"
Oct 30 19:37:46.408921 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:46.408883    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8f2440f-1238-46f6-ba78-90f9c38b574d-workload-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9\" (UID: \"e8f2440f-1238-46f6-ba78-90f9c38b574d\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9"
Oct 30 19:37:46.411030 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:46.411004    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8f2440f-1238-46f6-ba78-90f9c38b574d-istio-envoy\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9\" (UID: \"e8f2440f-1238-46f6-ba78-90f9c38b574d\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9"
Oct 30 19:37:46.411906 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:46.411886    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8f2440f-1238-46f6-ba78-90f9c38b574d-istio-podinfo\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9\" (UID: \"e8f2440f-1238-46f6-ba78-90f9c38b574d\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9"
Oct 30 19:37:46.417631 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:46.417605    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8f2440f-1238-46f6-ba78-90f9c38b574d-istio-token\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9\" (UID: \"e8f2440f-1238-46f6-ba78-90f9c38b574d\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9"
Oct 30 19:37:46.421832 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:46.421801    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6w8l\" (UniqueName: \"kubernetes.io/projected/e8f2440f-1238-46f6-ba78-90f9c38b574d-kube-api-access-n6w8l\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9\" (UID: \"e8f2440f-1238-46f6-ba78-90f9c38b574d\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9"
Oct 30 19:37:46.476038 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:46.476000    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9"
Oct 30 19:37:46.773373 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:37:46.773342    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8f2440f_1238_46f6_ba78_90f9c38b574d.slice/crio-00036b6651de3a17527070a9d27088be699730eb3f5042b7ecb4448acd91d08d WatchSource:0}: Error finding container 00036b6651de3a17527070a9d27088be699730eb3f5042b7ecb4448acd91d08d: Status 404 returned error can't find the container with id 00036b6651de3a17527070a9d27088be699730eb3f5042b7ecb4448acd91d08d
Oct 30 19:37:46.774543 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:46.774514    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9"]
Oct 30 19:37:46.777257 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:46.777214    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:37:46.777349 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:46.777311    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:37:46.777385 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:46.777353    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:37:47.092944 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:47.092840    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9" event={"ID":"e8f2440f-1238-46f6-ba78-90f9c38b574d","Type":"ContainerStarted","Data":"6aa0c38ba63755a080259cb76d2111e35ee8be8bbf1da42d55784ed8b17ac664"}
Oct 30 19:37:47.092944 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:47.092915    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9" event={"ID":"e8f2440f-1238-46f6-ba78-90f9c38b574d","Type":"ContainerStarted","Data":"00036b6651de3a17527070a9d27088be699730eb3f5042b7ecb4448acd91d08d"}
Oct 30 19:37:47.121297 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:47.121245    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9" podStartSLOduration=1.121230208 podStartE2EDuration="1.121230208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:47.118191987 +0000 UTC m=+1286.629251955" watchObservedRunningTime="2026-10-30 19:37:47.121230208 +0000 UTC m=+1286.632290172"
Oct 30 19:37:47.476784 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:47.476752    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9"
Oct 30 19:37:47.478729 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:47.478699    5043 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Oct 30 19:37:47.478890 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:47.478776    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9" podUID="e8f2440f-1238-46f6-ba78-90f9c38b574d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Oct 30 19:37:48.504537 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:48.489223    5043 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Oct 30 19:37:48.504537 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:48.489283    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9" podUID="e8f2440f-1238-46f6-ba78-90f9c38b574d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Oct 30 19:37:49.479237 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:49.479198    5043 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Oct 30 19:37:49.479543 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:49.479514    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9" podUID="e8f2440f-1238-46f6-ba78-90f9c38b574d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Oct 30 19:37:50.145455 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:50.145409    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd"]
Oct 30 19:37:50.152092 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:50.152063    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd"
Oct 30 19:37:50.208017 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:50.205777    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-156838-openshift-default-dockercfg-h9csk\""
Oct 30 19:37:50.218584 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:50.218552    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-istio-envoy\") pod \"e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd\" (UID: \"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5\") " pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd"
Oct 30 19:37:50.218870 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:50.218848    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-istio-podinfo\") pod \"e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd\" (UID: \"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5\") " pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd"
Oct 30 19:37:50.219435 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:50.219415    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-istio-token\") pod \"e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd\" (UID: \"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5\") " pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd"
Oct 30 19:37:50.219579 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:50.219564    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-workload-certs\") pod \"e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd\" (UID: \"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5\") " pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd"
Oct 30 19:37:50.219680 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:50.219666    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-istiod-ca-cert\") pod \"e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd\" (UID: \"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5\") " pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd"
Oct 30 19:37:50.219818 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:50.219800    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-workload-socket\") pod \"e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd\" (UID: \"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5\") " pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd"
Oct 30 19:37:50.219936 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:50.219922    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-istio-data\") pod \"e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd\" (UID: \"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5\") " pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd"
Oct 30 19:37:50.220162 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:50.220144    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkdwb\" (UniqueName: \"kubernetes.io/projected/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-kube-api-access-kkdwb\") pod \"e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd\" (UID: \"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5\") " pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd"
Oct 30 19:37:50.220491 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:50.220467    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-credential-socket\") pod \"e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd\" (UID: \"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5\") " pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd"
Oct 30 19:37:50.227213 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:50.226604    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd"]
Oct 30 19:37:50.323764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:50.323591    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-istio-podinfo\") pod \"e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd\" (UID: \"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5\") " pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd"
Oct 30 19:37:50.323764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:50.323720    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-istio-token\") pod \"e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd\" (UID: \"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5\") " pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd"
Oct 30 19:37:50.324039 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:50.323794    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-workload-certs\") pod \"e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd\" (UID: \"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5\") " pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd"
Oct 30 19:37:50.324039 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:50.323832    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-istiod-ca-cert\") pod \"e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd\" (UID: \"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5\") " pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd"
Oct 30 19:37:50.324039 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:50.323871    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-workload-socket\") pod \"e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd\" (UID: \"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5\") " pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd"
Oct 30 19:37:50.324039 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:50.323917    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-istio-data\") pod \"e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd\" (UID: \"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5\") " pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd"
Oct 30 19:37:50.324039 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:50.324024    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkdwb\" (UniqueName: \"kubernetes.io/projected/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-kube-api-access-kkdwb\") pod \"e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd\" (UID: \"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5\") " pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd"
Oct 30 19:37:50.324313 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:50.324086    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-credential-socket\") pod \"e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd\" (UID: \"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5\") " pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd"
Oct 30 19:37:50.324377 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:50.324314    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-istio-envoy\") pod \"e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd\" (UID: \"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5\") " pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd"
Oct 30 19:37:50.365866 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:50.342969    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-istio-envoy\") pod \"e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd\" (UID: \"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5\") " pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd"
Oct 30 19:37:50.365866 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:50.344576    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-workload-socket\") pod \"e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd\" (UID: \"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5\") " pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd"
Oct 30 19:37:50.365866 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:50.346070    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-istio-podinfo\") pod \"e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd\" (UID: \"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5\") " pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd"
Oct 30 19:37:50.365866 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:50.346526    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-istio-data\") pod \"e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd\" (UID: \"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5\") " pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd"
Oct 30 19:37:50.365866 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:50.346929    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-credential-socket\") pod \"e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd\" (UID: \"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5\") " pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd"
Oct 30 19:37:50.365866 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:50.347318    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-workload-certs\") pod \"e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd\" (UID: \"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5\") " pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd"
Oct 30 19:37:50.365866 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:50.347935    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-istiod-ca-cert\") pod \"e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd\" (UID: \"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5\") " pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd"
Oct 30 19:37:50.491513 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:50.489096    5043 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Oct 30 19:37:50.491513 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:50.489174    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9" podUID="e8f2440f-1238-46f6-ba78-90f9c38b574d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Oct 30 19:37:50.495772 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:50.494854    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-istio-token\") pod \"e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd\" (UID: \"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5\") " pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd"
Oct 30 19:37:50.521000 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:50.516573    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkdwb\" (UniqueName: \"kubernetes.io/projected/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-kube-api-access-kkdwb\") pod \"e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd\" (UID: \"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5\") " pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd"
Oct 30 19:37:50.528284 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:50.528225    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd"
Oct 30 19:37:51.104511 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:37:51.104446    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2db29833_fd7c_4dfd_88ba_6ae264d9f8b5.slice/crio-d28779c9c320dd14dabb5253a1f4063ae57cb8289e4a0459019c2b35567247e3 WatchSource:0}: Error finding container d28779c9c320dd14dabb5253a1f4063ae57cb8289e4a0459019c2b35567247e3: Status 404 returned error can't find the container with id d28779c9c320dd14dabb5253a1f4063ae57cb8289e4a0459019c2b35567247e3
Oct 30 19:37:51.107595 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:51.107556    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:37:51.107726 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:51.107661    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:37:51.107726 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:51.107707    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:37:51.110716 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:51.110695    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd"]
Oct 30 19:37:51.245808 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:51.245773    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd" event={"ID":"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5","Type":"ContainerStarted","Data":"bf77081744284b6717031a6400821be62368290fa84e5f36bf4da82dcffa73b4"}
Oct 30 19:37:51.246282 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:51.245835    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd" event={"ID":"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5","Type":"ContainerStarted","Data":"d28779c9c320dd14dabb5253a1f4063ae57cb8289e4a0459019c2b35567247e3"}
Oct 30 19:37:51.272626 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:51.272569    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd" podStartSLOduration=1.272548905 podStartE2EDuration="1.272548905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:51.271407407 +0000 UTC m=+1290.782467374" watchObservedRunningTime="2026-10-30 19:37:51.272548905 +0000 UTC m=+1290.783608873"
Oct 30 19:37:51.483761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:51.477179    5043 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Oct 30 19:37:51.483761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:51.477238    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9" podUID="e8f2440f-1238-46f6-ba78-90f9c38b574d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Oct 30 19:37:51.530877 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:51.530709    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd"
Oct 30 19:37:52.480181 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:52.480140    5043 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Oct 30 19:37:52.480713 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:52.480218    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9" podUID="e8f2440f-1238-46f6-ba78-90f9c38b574d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Oct 30 19:37:52.504520 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:52.504480    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6a6348-46bdl"]
Oct 30 19:37:52.514842 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:52.514811    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6a6348-46bdl"
Oct 30 19:37:52.519896 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:52.519867    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6a6348-46bdl"]
Oct 30 19:37:52.520348 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:52.520332    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:37:52.521182 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:52.521143    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:37:52.528853 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:52.528820    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:37:52.530011 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:52.529185    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:37:52.538097 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:52.538077    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd"
Oct 30 19:37:52.617341 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:52.617314    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9"]
Oct 30 19:37:52.617876 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:52.617842    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9" podUID="e8f2440f-1238-46f6-ba78-90f9c38b574d" containerName="istio-proxy" containerID="cri-o://6aa0c38ba63755a080259cb76d2111e35ee8be8bbf1da42d55784ed8b17ac664" gracePeriod=30
Oct 30 19:37:52.621486 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:52.620897    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/869b0618-5974-4bc0-91cf-339f483d115f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6a6348-46bdl\" (UID: \"869b0618-5974-4bc0-91cf-339f483d115f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6a6348-46bdl"
Oct 30 19:37:52.621486 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:52.621068    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/869b0618-5974-4bc0-91cf-339f483d115f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6a6348-46bdl\" (UID: \"869b0618-5974-4bc0-91cf-339f483d115f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6a6348-46bdl"
Oct 30 19:37:52.621486 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:52.621297    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p58j\" (UniqueName: \"kubernetes.io/projected/869b0618-5974-4bc0-91cf-339f483d115f-kube-api-access-4p58j\") pod \"maas-api-revoke-keys-e2e-api-b-6a6348-46bdl\" (UID: \"869b0618-5974-4bc0-91cf-339f483d115f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6a6348-46bdl"
Oct 30 19:37:52.623545 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:52.623505    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk"]
Oct 30 19:37:52.623801 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:52.623706    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk" podUID="6ae382e3-be21-4e69-bf70-a8c0b8517789" containerName="istio-proxy" containerID="cri-o://d0f492c58f63fcfc184b1ecf6d0a1fe3c10c76f34de4f2a24f54e62573e9b564" gracePeriod=30
Oct 30 19:37:52.629102 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:52.629068    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp"]
Oct 30 19:37:52.629265 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:52.629231    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp" podUID="dfd35ad6-36ee-44ba-8d74-021c61177b3b" containerName="istio-proxy" containerID="cri-o://94bfaaa913474ad26a70e390d1e55ba75d2433a5d84c825623a155189b77ad4c" gracePeriod=30
Oct 30 19:37:52.731909 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:52.728849    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p58j\" (UniqueName: \"kubernetes.io/projected/869b0618-5974-4bc0-91cf-339f483d115f-kube-api-access-4p58j\") pod \"maas-api-revoke-keys-e2e-api-b-6a6348-46bdl\" (UID: \"869b0618-5974-4bc0-91cf-339f483d115f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6a6348-46bdl"
Oct 30 19:37:52.731909 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:52.729084    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/869b0618-5974-4bc0-91cf-339f483d115f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6a6348-46bdl\" (UID: \"869b0618-5974-4bc0-91cf-339f483d115f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6a6348-46bdl"
Oct 30 19:37:52.731909 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:52.729292    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/869b0618-5974-4bc0-91cf-339f483d115f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6a6348-46bdl\" (UID: \"869b0618-5974-4bc0-91cf-339f483d115f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6a6348-46bdl"
Oct 30 19:37:52.740762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:52.736721    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/869b0618-5974-4bc0-91cf-339f483d115f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6a6348-46bdl\" (UID: \"869b0618-5974-4bc0-91cf-339f483d115f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6a6348-46bdl"
Oct 30 19:37:52.764757 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:52.758172    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/869b0618-5974-4bc0-91cf-339f483d115f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6a6348-46bdl\" (UID: \"869b0618-5974-4bc0-91cf-339f483d115f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6a6348-46bdl"
Oct 30 19:37:52.862436 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:52.862400    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p58j\" (UniqueName: \"kubernetes.io/projected/869b0618-5974-4bc0-91cf-339f483d115f-kube-api-access-4p58j\") pod \"maas-api-revoke-keys-e2e-api-b-6a6348-46bdl\" (UID: \"869b0618-5974-4bc0-91cf-339f483d115f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6a6348-46bdl"
Oct 30 19:37:53.145209 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:53.145134    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6a6348-46bdl"
Oct 30 19:37:53.404473 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:53.404336    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd"
Oct 30 19:37:53.431783 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:53.411988    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd"
Oct 30 19:37:53.889332 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:53.883555    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6a6348-46bdl"]
Oct 30 19:37:54.088253 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:54.085500    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-579ddb-6756775486-ghzlj"
Oct 30 19:37:54.088781 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:54.088761    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-579ddb-9b6756cf9-gfdfg"
Oct 30 19:37:54.504947 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:54.504891    5043 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:37:54.505264 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:54.505217    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk" podUID="6ae382e3-be21-4e69-bf70-a8c0b8517789" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:37:54.581820 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:54.581351    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6a6348-46bdl" event={"ID":"869b0618-5974-4bc0-91cf-339f483d115f","Type":"ContainerStarted","Data":"7cbd8e4f7ae9426b36adde78bb5c73550e3805304a23575f08fff5379d05ef84"}
Oct 30 19:37:55.042484 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:37:55.042449    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:37:55.595792 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:55.595705    5043 generic.go:356] "Generic (PLEG): container finished" podID="869b0618-5974-4bc0-91cf-339f483d115f" containerID="3ee0767c72ba5ca12fd92f11e4faf087f217ae19409366835f6b277951c2fabf" exitCode=0
Oct 30 19:37:55.596126 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:55.596094    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6a6348-46bdl" event={"ID":"869b0618-5974-4bc0-91cf-339f483d115f","Type":"ContainerDied","Data":"3ee0767c72ba5ca12fd92f11e4faf087f217ae19409366835f6b277951c2fabf"}
Oct 30 19:37:55.750399 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:55.750368    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-156838-79f475dcd-x2qg9"]
Oct 30 19:37:55.755320 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:55.755298    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-156838-79f475dcd-x2qg9"
Oct 30 19:37:55.763412 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:55.762908    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-156838\""
Oct 30 19:37:55.766405 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:55.765940    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-156838-dockercfg-k5kf5\""
Oct 30 19:37:55.769792 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:55.769768    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-156838-79f475dcd-x2qg9"]
Oct 30 19:37:55.827899 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:55.827862    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-156838-6749758ddc-tnv4d"]
Oct 30 19:37:55.842273 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:55.840958    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-156838-6749758ddc-tnv4d"]
Oct 30 19:37:55.842273 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:55.841061    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-156838-6749758ddc-tnv4d"
Oct 30 19:37:55.850955 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:55.850884    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9db7ea84-934d-41d6-aa27-c163dfc862c5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-156838-79f475dcd-x2qg9\" (UID: \"9db7ea84-934d-41d6-aa27-c163dfc862c5\") " pod="openshift-ingress/payload-processing-e2e-api-b-156838-79f475dcd-x2qg9"
Oct 30 19:37:55.851105 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:55.851052    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjvrm\" (UniqueName: \"kubernetes.io/projected/9db7ea84-934d-41d6-aa27-c163dfc862c5-kube-api-access-tjvrm\") pod \"payload-processing-e2e-api-b-156838-79f475dcd-x2qg9\" (UID: \"9db7ea84-934d-41d6-aa27-c163dfc862c5\") " pod="openshift-ingress/payload-processing-e2e-api-b-156838-79f475dcd-x2qg9"
Oct 30 19:37:55.851297 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:55.851271    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21041d00-c41b-4b80-839f-c48c0f6988c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-156838-6749758ddc-tnv4d\" (UID: \"21041d00-c41b-4b80-839f-c48c0f6988c7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-156838-6749758ddc-tnv4d"
Oct 30 19:37:55.851479 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:55.851457    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9hmf\" (UniqueName: \"kubernetes.io/projected/21041d00-c41b-4b80-839f-c48c0f6988c7-kube-api-access-m9hmf\") pod \"payload-pre-processing-e2e-api-b-156838-6749758ddc-tnv4d\" (UID: \"21041d00-c41b-4b80-839f-c48c0f6988c7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-156838-6749758ddc-tnv4d"
Oct 30 19:37:55.919359 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:55.919306    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp"]
Oct 30 19:37:55.924221 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:55.924193    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp"
Oct 30 19:37:55.933198 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:55.933174    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp"]
Oct 30 19:37:55.950341 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:55.950317    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-d1d256-openshift-default-dockercfg-jxksx\""
Oct 30 19:37:55.954202 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:55.954134    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjvrm\" (UniqueName: \"kubernetes.io/projected/9db7ea84-934d-41d6-aa27-c163dfc862c5-kube-api-access-tjvrm\") pod \"payload-processing-e2e-api-b-156838-79f475dcd-x2qg9\" (UID: \"9db7ea84-934d-41d6-aa27-c163dfc862c5\") " pod="openshift-ingress/payload-processing-e2e-api-b-156838-79f475dcd-x2qg9"
Oct 30 19:37:55.954368 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:55.954314    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74dccf1-3de7-4f70-a549-281361992615-credential-socket\") pod \"e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp\" (UID: \"a74dccf1-3de7-4f70-a549-281361992615\") " pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp"
Oct 30 19:37:55.954475 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:55.954385    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a74dccf1-3de7-4f70-a549-281361992615-istio-data\") pod \"e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp\" (UID: \"a74dccf1-3de7-4f70-a549-281361992615\") " pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp"
Oct 30 19:37:55.954578 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:55.954554    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74dccf1-3de7-4f70-a549-281361992615-workload-socket\") pod \"e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp\" (UID: \"a74dccf1-3de7-4f70-a549-281361992615\") " pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp"
Oct 30 19:37:55.954653 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:55.954635    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21041d00-c41b-4b80-839f-c48c0f6988c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-156838-6749758ddc-tnv4d\" (UID: \"21041d00-c41b-4b80-839f-c48c0f6988c7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-156838-6749758ddc-tnv4d"
Oct 30 19:37:55.954709 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:55.954668    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a74dccf1-3de7-4f70-a549-281361992615-istiod-ca-cert\") pod \"e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp\" (UID: \"a74dccf1-3de7-4f70-a549-281361992615\") " pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp"
Oct 30 19:37:55.954834 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:55.954815    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a74dccf1-3de7-4f70-a549-281361992615-workload-certs\") pod \"e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp\" (UID: \"a74dccf1-3de7-4f70-a549-281361992615\") " pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp"
Oct 30 19:37:55.954903 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:55.954873    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a74dccf1-3de7-4f70-a549-281361992615-istio-envoy\") pod \"e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp\" (UID: \"a74dccf1-3de7-4f70-a549-281361992615\") " pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp"
Oct 30 19:37:55.954957 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:55.954900    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a74dccf1-3de7-4f70-a549-281361992615-istio-token\") pod \"e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp\" (UID: \"a74dccf1-3de7-4f70-a549-281361992615\") " pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp"
Oct 30 19:37:55.954957 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:55.954931    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9hmf\" (UniqueName: \"kubernetes.io/projected/21041d00-c41b-4b80-839f-c48c0f6988c7-kube-api-access-m9hmf\") pod \"payload-pre-processing-e2e-api-b-156838-6749758ddc-tnv4d\" (UID: \"21041d00-c41b-4b80-839f-c48c0f6988c7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-156838-6749758ddc-tnv4d"
Oct 30 19:37:55.955118 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:55.955101    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9db7ea84-934d-41d6-aa27-c163dfc862c5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-156838-79f475dcd-x2qg9\" (UID: \"9db7ea84-934d-41d6-aa27-c163dfc862c5\") " pod="openshift-ingress/payload-processing-e2e-api-b-156838-79f475dcd-x2qg9"
Oct 30 19:37:55.955177 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:55.955145    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a74dccf1-3de7-4f70-a549-281361992615-istio-podinfo\") pod \"e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp\" (UID: \"a74dccf1-3de7-4f70-a549-281361992615\") " pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp"
Oct 30 19:37:55.955242 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:55.955223    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sphlb\" (UniqueName: \"kubernetes.io/projected/a74dccf1-3de7-4f70-a549-281361992615-kube-api-access-sphlb\") pod \"e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp\" (UID: \"a74dccf1-3de7-4f70-a549-281361992615\") " pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp"
Oct 30 19:37:55.956465 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:55.956298    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21041d00-c41b-4b80-839f-c48c0f6988c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-156838-6749758ddc-tnv4d\" (UID: \"21041d00-c41b-4b80-839f-c48c0f6988c7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-156838-6749758ddc-tnv4d"
Oct 30 19:37:55.956812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:55.956791    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9db7ea84-934d-41d6-aa27-c163dfc862c5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-156838-79f475dcd-x2qg9\" (UID: \"9db7ea84-934d-41d6-aa27-c163dfc862c5\") " pod="openshift-ingress/payload-processing-e2e-api-b-156838-79f475dcd-x2qg9"
Oct 30 19:37:55.997759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:55.991405    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjvrm\" (UniqueName: \"kubernetes.io/projected/9db7ea84-934d-41d6-aa27-c163dfc862c5-kube-api-access-tjvrm\") pod \"payload-processing-e2e-api-b-156838-79f475dcd-x2qg9\" (UID: \"9db7ea84-934d-41d6-aa27-c163dfc862c5\") " pod="openshift-ingress/payload-processing-e2e-api-b-156838-79f475dcd-x2qg9"
Oct 30 19:37:56.063279 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:56.042578    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9hmf\" (UniqueName: \"kubernetes.io/projected/21041d00-c41b-4b80-839f-c48c0f6988c7-kube-api-access-m9hmf\") pod \"payload-pre-processing-e2e-api-b-156838-6749758ddc-tnv4d\" (UID: \"21041d00-c41b-4b80-839f-c48c0f6988c7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-156838-6749758ddc-tnv4d"
Oct 30 19:37:56.063279 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:56.060393    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a74dccf1-3de7-4f70-a549-281361992615-istiod-ca-cert\") pod \"e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp\" (UID: \"a74dccf1-3de7-4f70-a549-281361992615\") " pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp"
Oct 30 19:37:56.063279 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:56.060448    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a74dccf1-3de7-4f70-a549-281361992615-workload-certs\") pod \"e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp\" (UID: \"a74dccf1-3de7-4f70-a549-281361992615\") " pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp"
Oct 30 19:37:56.063279 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:56.060485    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a74dccf1-3de7-4f70-a549-281361992615-istio-envoy\") pod \"e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp\" (UID: \"a74dccf1-3de7-4f70-a549-281361992615\") " pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp"
Oct 30 19:37:56.063279 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:56.060506    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a74dccf1-3de7-4f70-a549-281361992615-istio-token\") pod \"e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp\" (UID: \"a74dccf1-3de7-4f70-a549-281361992615\") " pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp"
Oct 30 19:37:56.063279 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:56.060624    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a74dccf1-3de7-4f70-a549-281361992615-istio-podinfo\") pod \"e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp\" (UID: \"a74dccf1-3de7-4f70-a549-281361992615\") " pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp"
Oct 30 19:37:56.063279 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:56.060677    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sphlb\" (UniqueName: \"kubernetes.io/projected/a74dccf1-3de7-4f70-a549-281361992615-kube-api-access-sphlb\") pod \"e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp\" (UID: \"a74dccf1-3de7-4f70-a549-281361992615\") " pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp"
Oct 30 19:37:56.063279 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:56.060784    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74dccf1-3de7-4f70-a549-281361992615-credential-socket\") pod \"e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp\" (UID: \"a74dccf1-3de7-4f70-a549-281361992615\") " pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp"
Oct 30 19:37:56.063279 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:56.060820    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a74dccf1-3de7-4f70-a549-281361992615-istio-data\") pod \"e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp\" (UID: \"a74dccf1-3de7-4f70-a549-281361992615\") " pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp"
Oct 30 19:37:56.063279 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:56.060893    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74dccf1-3de7-4f70-a549-281361992615-workload-socket\") pod \"e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp\" (UID: \"a74dccf1-3de7-4f70-a549-281361992615\") " pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp"
Oct 30 19:37:56.063279 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:56.061326    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74dccf1-3de7-4f70-a549-281361992615-workload-socket\") pod \"e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp\" (UID: \"a74dccf1-3de7-4f70-a549-281361992615\") " pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp"
Oct 30 19:37:56.063279 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:56.061925    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a74dccf1-3de7-4f70-a549-281361992615-istiod-ca-cert\") pod \"e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp\" (UID: \"a74dccf1-3de7-4f70-a549-281361992615\") " pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp"
Oct 30 19:37:56.063279 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:56.062142    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a74dccf1-3de7-4f70-a549-281361992615-workload-certs\") pod \"e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp\" (UID: \"a74dccf1-3de7-4f70-a549-281361992615\") " pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp"
Oct 30 19:37:56.101184 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:56.101106    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74dccf1-3de7-4f70-a549-281361992615-credential-socket\") pod \"e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp\" (UID: \"a74dccf1-3de7-4f70-a549-281361992615\") " pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp"
Oct 30 19:37:56.101708 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:56.101683    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a74dccf1-3de7-4f70-a549-281361992615-istio-data\") pod \"e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp\" (UID: \"a74dccf1-3de7-4f70-a549-281361992615\") " pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp"
Oct 30 19:37:56.103119 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:56.103074    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-156838-79f475dcd-x2qg9"
Oct 30 19:37:56.135879 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:56.135832    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a74dccf1-3de7-4f70-a549-281361992615-istio-token\") pod \"e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp\" (UID: \"a74dccf1-3de7-4f70-a549-281361992615\") " pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp"
Oct 30 19:37:56.185776 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:56.156075    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a74dccf1-3de7-4f70-a549-281361992615-istio-envoy\") pod \"e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp\" (UID: \"a74dccf1-3de7-4f70-a549-281361992615\") " pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp"
Oct 30 19:37:56.339565 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:56.190494    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sphlb\" (UniqueName: \"kubernetes.io/projected/a74dccf1-3de7-4f70-a549-281361992615-kube-api-access-sphlb\") pod \"e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp\" (UID: \"a74dccf1-3de7-4f70-a549-281361992615\") " pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp"
Oct 30 19:37:56.339565 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:56.198993    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-156838-6749758ddc-tnv4d"
Oct 30 19:37:56.339565 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:56.241665    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a74dccf1-3de7-4f70-a549-281361992615-istio-podinfo\") pod \"e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp\" (UID: \"a74dccf1-3de7-4f70-a549-281361992615\") " pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp"
Oct 30 19:37:56.339565 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:56.256314    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp"
Oct 30 19:37:56.678125 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:56.677044    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-156838-79f475dcd-x2qg9"]
Oct 30 19:37:56.683442 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:37:56.683410    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9db7ea84_934d_41d6_aa27_c163dfc862c5.slice/crio-873a0fbeb8c947a048c7b97a9b025ae5406b4c3833fb527d00479a0a6cd591b2 WatchSource:0}: Error finding container 873a0fbeb8c947a048c7b97a9b025ae5406b4c3833fb527d00479a0a6cd591b2: Status 404 returned error can't find the container with id 873a0fbeb8c947a048c7b97a9b025ae5406b4c3833fb527d00479a0a6cd591b2
Oct 30 19:37:56.733981 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:56.732887    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp"]
Oct 30 19:37:56.745671 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:56.745627    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:37:56.745972 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:56.745949    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:37:56.746106 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:56.746091    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:37:56.917030 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:56.913955    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6a6348-46bdl"
Oct 30 19:37:57.027583 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:37:57.027523    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21041d00_c41b_4b80_839f_c48c0f6988c7.slice/crio-14cd2ba10157eb798f3d20c6c794a75cdc5bb0fccb119e6c0e0356274e2b0c6a WatchSource:0}: Error finding container 14cd2ba10157eb798f3d20c6c794a75cdc5bb0fccb119e6c0e0356274e2b0c6a: Status 404 returned error can't find the container with id 14cd2ba10157eb798f3d20c6c794a75cdc5bb0fccb119e6c0e0356274e2b0c6a
Oct 30 19:37:57.036334 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:57.035057    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-156838-6749758ddc-tnv4d"]
Oct 30 19:37:57.056981 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:57.056950    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/869b0618-5974-4bc0-91cf-339f483d115f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/869b0618-5974-4bc0-91cf-339f483d115f-maas-api-openshift-ca\") pod \"869b0618-5974-4bc0-91cf-339f483d115f\" (UID: \"869b0618-5974-4bc0-91cf-339f483d115f\") "
Oct 30 19:37:57.057268 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:57.057250    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/869b0618-5974-4bc0-91cf-339f483d115f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/869b0618-5974-4bc0-91cf-339f483d115f-maas-api-xks-ca\") pod \"869b0618-5974-4bc0-91cf-339f483d115f\" (UID: \"869b0618-5974-4bc0-91cf-339f483d115f\") "
Oct 30 19:37:57.057359 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:57.057344    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/869b0618-5974-4bc0-91cf-339f483d115f-kube-api-access-4p58j\" (UniqueName: \"kubernetes.io/projected/869b0618-5974-4bc0-91cf-339f483d115f-kube-api-access-4p58j\") pod \"869b0618-5974-4bc0-91cf-339f483d115f\" (UID: \"869b0618-5974-4bc0-91cf-339f483d115f\") "
Oct 30 19:37:57.059400 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:57.059372    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/869b0618-5974-4bc0-91cf-339f483d115f-maas-api-openshift-ca" pod "869b0618-5974-4bc0-91cf-339f483d115f" (UID: "869b0618-5974-4bc0-91cf-339f483d115f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:37:57.062015 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:57.061985    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/869b0618-5974-4bc0-91cf-339f483d115f-maas-api-xks-ca" pod "869b0618-5974-4bc0-91cf-339f483d115f" (UID: "869b0618-5974-4bc0-91cf-339f483d115f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:37:57.062114 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:57.062030    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/869b0618-5974-4bc0-91cf-339f483d115f-kube-api-access-4p58j" pod "869b0618-5974-4bc0-91cf-339f483d115f" (UID: "869b0618-5974-4bc0-91cf-339f483d115f"). InnerVolumeSpecName "kube-api-access-4p58j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:57.159934 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:57.159894    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p58j\" (UniqueName: \"kubernetes.io/projected/869b0618-5974-4bc0-91cf-339f483d115f-kube-api-access-4p58j\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:57.159934 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:57.159937    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/869b0618-5974-4bc0-91cf-339f483d115f-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:57.160493 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:57.159954    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/869b0618-5974-4bc0-91cf-339f483d115f-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:57.776762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:57.772248    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-156838-6749758ddc-tnv4d" event={"ID":"21041d00-c41b-4b80-839f-c48c0f6988c7","Type":"ContainerStarted","Data":"21054e68e815c5ae4fd338a67895e5d5f3c58e0ef4d5c977521c9421d176a553"}
Oct 30 19:37:57.776762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:57.772322    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-156838-6749758ddc-tnv4d" event={"ID":"21041d00-c41b-4b80-839f-c48c0f6988c7","Type":"ContainerStarted","Data":"14cd2ba10157eb798f3d20c6c794a75cdc5bb0fccb119e6c0e0356274e2b0c6a"}
Oct 30 19:37:57.776762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:57.772966    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-156838-6749758ddc-tnv4d"
Oct 30 19:37:57.776762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:57.776016    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-156838-79f475dcd-x2qg9" event={"ID":"9db7ea84-934d-41d6-aa27-c163dfc862c5","Type":"ContainerStarted","Data":"0581936c34a1300d468eeadbd7c5aa7cfe092626c6db1d6c943d30283cd7f9cb"}
Oct 30 19:37:57.776762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:57.776069    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-156838-79f475dcd-x2qg9" event={"ID":"9db7ea84-934d-41d6-aa27-c163dfc862c5","Type":"ContainerStarted","Data":"873a0fbeb8c947a048c7b97a9b025ae5406b4c3833fb527d00479a0a6cd591b2"}
Oct 30 19:37:57.776762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:57.776425    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-156838-79f475dcd-x2qg9"
Oct 30 19:37:57.783402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:57.782404    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6a6348-46bdl" event={"ID":"869b0618-5974-4bc0-91cf-339f483d115f","Type":"ContainerDied","Data":"7cbd8e4f7ae9426b36adde78bb5c73550e3805304a23575f08fff5379d05ef84"}
Oct 30 19:37:57.783402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:57.782463    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cbd8e4f7ae9426b36adde78bb5c73550e3805304a23575f08fff5379d05ef84"
Oct 30 19:37:57.783402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:57.782543    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6a6348-46bdl"
Oct 30 19:37:57.797401 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:57.797145    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp" event={"ID":"a74dccf1-3de7-4f70-a549-281361992615","Type":"ContainerStarted","Data":"27ac769c108e2a9c3268d15f324d53c35afa80c74599451c9d15b3ea126d1e9e"}
Oct 30 19:37:57.797401 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:57.797212    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp" event={"ID":"a74dccf1-3de7-4f70-a549-281361992615","Type":"ContainerStarted","Data":"9c40a602b3adc37a56963dd0f958fc71ee33d3ca829e427a5d9ebd661b8cbe3f"}
Oct 30 19:37:57.812779 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:57.810719    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-156838-6749758ddc-tnv4d" podStartSLOduration=2.810697032 podStartE2EDuration="2.810697032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:57.805672267 +0000 UTC m=+1297.316732233" watchObservedRunningTime="2026-10-30 19:37:57.810697032 +0000 UTC m=+1297.321756997"
Oct 30 19:37:57.937217 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:57.937078    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-156838-79f475dcd-x2qg9" podStartSLOduration=2.937056859 podStartE2EDuration="2.937056859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:57.914786868 +0000 UTC m=+1297.425846834" watchObservedRunningTime="2026-10-30 19:37:57.937056859 +0000 UTC m=+1297.448116832"
Oct 30 19:37:57.938712 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:57.938660    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp" podStartSLOduration=2.938645506 podStartE2EDuration="2.938645506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:57.866426657 +0000 UTC m=+1297.377486623" watchObservedRunningTime="2026-10-30 19:37:57.938645506 +0000 UTC m=+1297.449705473"
Oct 30 19:37:58.271896 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.270910    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp"
Oct 30 19:37:58.321483 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.321447    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk"
Oct 30 19:37:58.327996 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.327962    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp"
Oct 30 19:37:58.337796 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.337707    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp"
Oct 30 19:37:58.418749 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.418708    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfd35ad6-36ee-44ba-8d74-021c61177b3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfd35ad6-36ee-44ba-8d74-021c61177b3b-istio-data\") pod \"dfd35ad6-36ee-44ba-8d74-021c61177b3b\" (UID: \"dfd35ad6-36ee-44ba-8d74-021c61177b3b\") "
Oct 30 19:37:58.419002 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.418815    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfd35ad6-36ee-44ba-8d74-021c61177b3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd35ad6-36ee-44ba-8d74-021c61177b3b-workload-socket\") pod \"dfd35ad6-36ee-44ba-8d74-021c61177b3b\" (UID: \"dfd35ad6-36ee-44ba-8d74-021c61177b3b\") "
Oct 30 19:37:58.419002 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.418907    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ae382e3-be21-4e69-bf70-a8c0b8517789-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ae382e3-be21-4e69-bf70-a8c0b8517789-istio-envoy\") pod \"6ae382e3-be21-4e69-bf70-a8c0b8517789\" (UID: \"6ae382e3-be21-4e69-bf70-a8c0b8517789\") "
Oct 30 19:37:58.419002 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.418946    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ae382e3-be21-4e69-bf70-a8c0b8517789-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ae382e3-be21-4e69-bf70-a8c0b8517789-istio-podinfo\") pod \"6ae382e3-be21-4e69-bf70-a8c0b8517789\" (UID: \"6ae382e3-be21-4e69-bf70-a8c0b8517789\") "
Oct 30 19:37:58.419253 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.419004    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ae382e3-be21-4e69-bf70-a8c0b8517789-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ae382e3-be21-4e69-bf70-a8c0b8517789-workload-certs\") pod \"6ae382e3-be21-4e69-bf70-a8c0b8517789\" (UID: \"6ae382e3-be21-4e69-bf70-a8c0b8517789\") "
Oct 30 19:37:58.419253 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.419046    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ae382e3-be21-4e69-bf70-a8c0b8517789-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae382e3-be21-4e69-bf70-a8c0b8517789-workload-socket\") pod \"6ae382e3-be21-4e69-bf70-a8c0b8517789\" (UID: \"6ae382e3-be21-4e69-bf70-a8c0b8517789\") "
Oct 30 19:37:58.419253 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.419075    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ae382e3-be21-4e69-bf70-a8c0b8517789-istio-token\" (UniqueName: \"kubernetes.io/projected/6ae382e3-be21-4e69-bf70-a8c0b8517789-istio-token\") pod \"6ae382e3-be21-4e69-bf70-a8c0b8517789\" (UID: \"6ae382e3-be21-4e69-bf70-a8c0b8517789\") "
Oct 30 19:37:58.419253 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.419184    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ae382e3-be21-4e69-bf70-a8c0b8517789-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae382e3-be21-4e69-bf70-a8c0b8517789-credential-socket\") pod \"6ae382e3-be21-4e69-bf70-a8c0b8517789\" (UID: \"6ae382e3-be21-4e69-bf70-a8c0b8517789\") "
Oct 30 19:37:58.419253 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.419221    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ae382e3-be21-4e69-bf70-a8c0b8517789-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ae382e3-be21-4e69-bf70-a8c0b8517789-istio-data\") pod \"6ae382e3-be21-4e69-bf70-a8c0b8517789\" (UID: \"6ae382e3-be21-4e69-bf70-a8c0b8517789\") "
Oct 30 19:37:58.419553 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.419257    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dfd35ad6-36ee-44ba-8d74-021c61177b3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfd35ad6-36ee-44ba-8d74-021c61177b3b-istio-podinfo\") pod \"dfd35ad6-36ee-44ba-8d74-021c61177b3b\" (UID: \"dfd35ad6-36ee-44ba-8d74-021c61177b3b\") "
Oct 30 19:37:58.419553 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.419335    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfd35ad6-36ee-44ba-8d74-021c61177b3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfd35ad6-36ee-44ba-8d74-021c61177b3b-istio-envoy\") pod \"dfd35ad6-36ee-44ba-8d74-021c61177b3b\" (UID: \"dfd35ad6-36ee-44ba-8d74-021c61177b3b\") "
Oct 30 19:37:58.419553 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.419038    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfd35ad6-36ee-44ba-8d74-021c61177b3b-istio-data" pod "dfd35ad6-36ee-44ba-8d74-021c61177b3b" (UID: "dfd35ad6-36ee-44ba-8d74-021c61177b3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:58.419553 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.419458    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfd35ad6-36ee-44ba-8d74-021c61177b3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfd35ad6-36ee-44ba-8d74-021c61177b3b-istiod-ca-cert\") pod \"dfd35ad6-36ee-44ba-8d74-021c61177b3b\" (UID: \"dfd35ad6-36ee-44ba-8d74-021c61177b3b\") "
Oct 30 19:37:58.419553 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.419489    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfd35ad6-36ee-44ba-8d74-021c61177b3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfd35ad6-36ee-44ba-8d74-021c61177b3b-workload-certs\") pod \"dfd35ad6-36ee-44ba-8d74-021c61177b3b\" (UID: \"dfd35ad6-36ee-44ba-8d74-021c61177b3b\") "
Oct 30 19:37:58.419922 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.419587    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfd35ad6-36ee-44ba-8d74-021c61177b3b-kube-api-access-vcgk2\" (UniqueName: \"kubernetes.io/projected/dfd35ad6-36ee-44ba-8d74-021c61177b3b-kube-api-access-vcgk2\") pod \"dfd35ad6-36ee-44ba-8d74-021c61177b3b\" (UID: \"dfd35ad6-36ee-44ba-8d74-021c61177b3b\") "
Oct 30 19:37:58.419922 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.419622    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae382e3-be21-4e69-bf70-a8c0b8517789-workload-socket" pod "6ae382e3-be21-4e69-bf70-a8c0b8517789" (UID: "6ae382e3-be21-4e69-bf70-a8c0b8517789"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:58.419922 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.419647    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ae382e3-be21-4e69-bf70-a8c0b8517789-kube-api-access-xmbk6\" (UniqueName: \"kubernetes.io/projected/6ae382e3-be21-4e69-bf70-a8c0b8517789-kube-api-access-xmbk6\") pod \"6ae382e3-be21-4e69-bf70-a8c0b8517789\" (UID: \"6ae382e3-be21-4e69-bf70-a8c0b8517789\") "
Oct 30 19:37:58.419922 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.419718    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfd35ad6-36ee-44ba-8d74-021c61177b3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd35ad6-36ee-44ba-8d74-021c61177b3b-credential-socket\") pod \"dfd35ad6-36ee-44ba-8d74-021c61177b3b\" (UID: \"dfd35ad6-36ee-44ba-8d74-021c61177b3b\") "
Oct 30 19:37:58.419922 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.419784    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae382e3-be21-4e69-bf70-a8c0b8517789-istio-data" pod "6ae382e3-be21-4e69-bf70-a8c0b8517789" (UID: "6ae382e3-be21-4e69-bf70-a8c0b8517789"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:58.419922 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.419816    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ae382e3-be21-4e69-bf70-a8c0b8517789-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ae382e3-be21-4e69-bf70-a8c0b8517789-istiod-ca-cert\") pod \"6ae382e3-be21-4e69-bf70-a8c0b8517789\" (UID: \"6ae382e3-be21-4e69-bf70-a8c0b8517789\") "
Oct 30 19:37:58.419922 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.419855    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfd35ad6-36ee-44ba-8d74-021c61177b3b-istio-token\" (UniqueName: \"kubernetes.io/projected/dfd35ad6-36ee-44ba-8d74-021c61177b3b-istio-token\") pod \"dfd35ad6-36ee-44ba-8d74-021c61177b3b\" (UID: \"dfd35ad6-36ee-44ba-8d74-021c61177b3b\") "
Oct 30 19:37:58.420484 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.420036    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae382e3-be21-4e69-bf70-a8c0b8517789-workload-certs" pod "6ae382e3-be21-4e69-bf70-a8c0b8517789" (UID: "6ae382e3-be21-4e69-bf70-a8c0b8517789"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:58.420792 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.420770    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfd35ad6-36ee-44ba-8d74-021c61177b3b-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:58.420882 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.420797    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ae382e3-be21-4e69-bf70-a8c0b8517789-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:58.420882 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.420813    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae382e3-be21-4e69-bf70-a8c0b8517789-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:58.420882 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.420827    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ae382e3-be21-4e69-bf70-a8c0b8517789-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:58.421371 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.421348    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfd35ad6-36ee-44ba-8d74-021c61177b3b-workload-socket" pod "dfd35ad6-36ee-44ba-8d74-021c61177b3b" (UID: "dfd35ad6-36ee-44ba-8d74-021c61177b3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:58.421559 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.421538    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfd35ad6-36ee-44ba-8d74-021c61177b3b-credential-socket" pod "dfd35ad6-36ee-44ba-8d74-021c61177b3b" (UID: "dfd35ad6-36ee-44ba-8d74-021c61177b3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:58.421826 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.421774    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae382e3-be21-4e69-bf70-a8c0b8517789-credential-socket" pod "6ae382e3-be21-4e69-bf70-a8c0b8517789" (UID: "6ae382e3-be21-4e69-bf70-a8c0b8517789"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:58.422072 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.422052    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ae382e3-be21-4e69-bf70-a8c0b8517789-istiod-ca-cert" pod "6ae382e3-be21-4e69-bf70-a8c0b8517789" (UID: "6ae382e3-be21-4e69-bf70-a8c0b8517789"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:37:58.422449 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.422391    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae382e3-be21-4e69-bf70-a8c0b8517789-istio-envoy" pod "6ae382e3-be21-4e69-bf70-a8c0b8517789" (UID: "6ae382e3-be21-4e69-bf70-a8c0b8517789"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:58.422449 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.422463    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ae382e3-be21-4e69-bf70-a8c0b8517789-istio-podinfo" pod "6ae382e3-be21-4e69-bf70-a8c0b8517789" (UID: "6ae382e3-be21-4e69-bf70-a8c0b8517789"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:37:58.422682 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.422620    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfd35ad6-36ee-44ba-8d74-021c61177b3b-workload-certs" pod "dfd35ad6-36ee-44ba-8d74-021c61177b3b" (UID: "dfd35ad6-36ee-44ba-8d74-021c61177b3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:58.422809 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.422785    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfd35ad6-36ee-44ba-8d74-021c61177b3b-istiod-ca-cert" pod "dfd35ad6-36ee-44ba-8d74-021c61177b3b" (UID: "dfd35ad6-36ee-44ba-8d74-021c61177b3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:37:58.424400 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.424367    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ae382e3-be21-4e69-bf70-a8c0b8517789-kube-api-access-xmbk6" pod "6ae382e3-be21-4e69-bf70-a8c0b8517789" (UID: "6ae382e3-be21-4e69-bf70-a8c0b8517789"). InnerVolumeSpecName "kube-api-access-xmbk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:58.425472 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.425443    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfd35ad6-36ee-44ba-8d74-021c61177b3b-istio-podinfo" pod "dfd35ad6-36ee-44ba-8d74-021c61177b3b" (UID: "dfd35ad6-36ee-44ba-8d74-021c61177b3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:37:58.425559 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.425541    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfd35ad6-36ee-44ba-8d74-021c61177b3b-istio-envoy" pod "dfd35ad6-36ee-44ba-8d74-021c61177b3b" (UID: "dfd35ad6-36ee-44ba-8d74-021c61177b3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:58.425634 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.425582    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfd35ad6-36ee-44ba-8d74-021c61177b3b-kube-api-access-vcgk2" pod "dfd35ad6-36ee-44ba-8d74-021c61177b3b" (UID: "dfd35ad6-36ee-44ba-8d74-021c61177b3b"). InnerVolumeSpecName "kube-api-access-vcgk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:58.425709 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.425692    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfd35ad6-36ee-44ba-8d74-021c61177b3b-istio-token" pod "dfd35ad6-36ee-44ba-8d74-021c61177b3b" (UID: "dfd35ad6-36ee-44ba-8d74-021c61177b3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:58.425908 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.425884    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ae382e3-be21-4e69-bf70-a8c0b8517789-istio-token" pod "6ae382e3-be21-4e69-bf70-a8c0b8517789" (UID: "6ae382e3-be21-4e69-bf70-a8c0b8517789"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:58.522155 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.522054    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfd35ad6-36ee-44ba-8d74-021c61177b3b-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:58.522155 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.522087    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfd35ad6-36ee-44ba-8d74-021c61177b3b-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:58.522155 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.522102    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcgk2\" (UniqueName: \"kubernetes.io/projected/dfd35ad6-36ee-44ba-8d74-021c61177b3b-kube-api-access-vcgk2\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:58.522155 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.522117    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmbk6\" (UniqueName: \"kubernetes.io/projected/6ae382e3-be21-4e69-bf70-a8c0b8517789-kube-api-access-xmbk6\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:58.522155 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.522130    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd35ad6-36ee-44ba-8d74-021c61177b3b-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:58.522155 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.522142    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ae382e3-be21-4e69-bf70-a8c0b8517789-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:58.522155 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.522154    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfd35ad6-36ee-44ba-8d74-021c61177b3b-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:58.522518 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.522167    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd35ad6-36ee-44ba-8d74-021c61177b3b-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:58.522518 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.522179    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ae382e3-be21-4e69-bf70-a8c0b8517789-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:58.522518 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.522190    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ae382e3-be21-4e69-bf70-a8c0b8517789-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:58.522518 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.522201    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ae382e3-be21-4e69-bf70-a8c0b8517789-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:58.522518 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.522213    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae382e3-be21-4e69-bf70-a8c0b8517789-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:58.522518 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.522226    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfd35ad6-36ee-44ba-8d74-021c61177b3b-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:58.522518 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.522237    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfd35ad6-36ee-44ba-8d74-021c61177b3b-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:58.823964 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.823856    5043 generic.go:356] "Generic (PLEG): container finished" podID="6ae382e3-be21-4e69-bf70-a8c0b8517789" containerID="d0f492c58f63fcfc184b1ecf6d0a1fe3c10c76f34de4f2a24f54e62573e9b564" exitCode=0
Oct 30 19:37:58.823964 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.823938    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk" event={"ID":"6ae382e3-be21-4e69-bf70-a8c0b8517789","Type":"ContainerDied","Data":"d0f492c58f63fcfc184b1ecf6d0a1fe3c10c76f34de4f2a24f54e62573e9b564"}
Oct 30 19:37:58.823964 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.823961    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk"
Oct 30 19:37:58.824264 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.824005    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk" event={"ID":"6ae382e3-be21-4e69-bf70-a8c0b8517789","Type":"ContainerDied","Data":"7ef9ee0386e42e0994294082c8014ff993bb3e400357e96170b803fa4eff9a35"}
Oct 30 19:37:58.824264 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.824035    5043 scope.go:122] "RemoveContainer" containerID="d0f492c58f63fcfc184b1ecf6d0a1fe3c10c76f34de4f2a24f54e62573e9b564"
Oct 30 19:37:58.830487 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.830463    5043 generic.go:356] "Generic (PLEG): container finished" podID="dfd35ad6-36ee-44ba-8d74-021c61177b3b" containerID="94bfaaa913474ad26a70e390d1e55ba75d2433a5d84c825623a155189b77ad4c" exitCode=0
Oct 30 19:37:58.830893 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.830693    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp"
Oct 30 19:37:58.830893 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.830711    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp" event={"ID":"dfd35ad6-36ee-44ba-8d74-021c61177b3b","Type":"ContainerDied","Data":"94bfaaa913474ad26a70e390d1e55ba75d2433a5d84c825623a155189b77ad4c"}
Oct 30 19:37:58.831058 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.830957    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp" event={"ID":"dfd35ad6-36ee-44ba-8d74-021c61177b3b","Type":"ContainerDied","Data":"ddb6fedb85e41b832d4c8a38e300c51523d63e97c4c6c712d63b3b6e4ff0ec40"}
Oct 30 19:37:58.831357 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.831337    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp"
Oct 30 19:37:58.835428 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.834810    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp"
Oct 30 19:37:58.862113 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.859061    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk"]
Oct 30 19:37:58.869992 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.866993    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-zcdmk"]
Oct 30 19:37:58.871788 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.871748    5043 scope.go:122] "RemoveContainer" containerID="d0f492c58f63fcfc184b1ecf6d0a1fe3c10c76f34de4f2a24f54e62573e9b564"
Oct 30 19:37:58.872244 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:37:58.872221    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0f492c58f63fcfc184b1ecf6d0a1fe3c10c76f34de4f2a24f54e62573e9b564\": container with ID starting with d0f492c58f63fcfc184b1ecf6d0a1fe3c10c76f34de4f2a24f54e62573e9b564 not found: ID does not exist" containerID="d0f492c58f63fcfc184b1ecf6d0a1fe3c10c76f34de4f2a24f54e62573e9b564"
Oct 30 19:37:58.872344 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.872252    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0f492c58f63fcfc184b1ecf6d0a1fe3c10c76f34de4f2a24f54e62573e9b564"} err="failed to get container status \"d0f492c58f63fcfc184b1ecf6d0a1fe3c10c76f34de4f2a24f54e62573e9b564\": rpc error: code = NotFound desc = could not find container \"d0f492c58f63fcfc184b1ecf6d0a1fe3c10c76f34de4f2a24f54e62573e9b564\": container with ID starting with d0f492c58f63fcfc184b1ecf6d0a1fe3c10c76f34de4f2a24f54e62573e9b564 not found: ID does not exist"
Oct 30 19:37:58.872344 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.872269    5043 scope.go:122] "RemoveContainer" containerID="94bfaaa913474ad26a70e390d1e55ba75d2433a5d84c825623a155189b77ad4c"
Oct 30 19:37:58.910138 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.909936    5043 scope.go:122] "RemoveContainer" containerID="b1d3d9f4c5634497a9f52cb9018117c2a4b9b81baf656cf5a88532f62f9eafec"
Oct 30 19:37:58.951857 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.951788    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp"]
Oct 30 19:37:58.955411 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.955382    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-bqssp"]
Oct 30 19:37:58.962406 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.962384    5043 scope.go:122] "RemoveContainer" containerID="94bfaaa913474ad26a70e390d1e55ba75d2433a5d84c825623a155189b77ad4c"
Oct 30 19:37:58.962855 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:37:58.962822    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94bfaaa913474ad26a70e390d1e55ba75d2433a5d84c825623a155189b77ad4c\": container with ID starting with 94bfaaa913474ad26a70e390d1e55ba75d2433a5d84c825623a155189b77ad4c not found: ID does not exist" containerID="94bfaaa913474ad26a70e390d1e55ba75d2433a5d84c825623a155189b77ad4c"
Oct 30 19:37:58.962968 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.962865    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94bfaaa913474ad26a70e390d1e55ba75d2433a5d84c825623a155189b77ad4c"} err="failed to get container status \"94bfaaa913474ad26a70e390d1e55ba75d2433a5d84c825623a155189b77ad4c\": rpc error: code = NotFound desc = could not find container \"94bfaaa913474ad26a70e390d1e55ba75d2433a5d84c825623a155189b77ad4c\": container with ID starting with 94bfaaa913474ad26a70e390d1e55ba75d2433a5d84c825623a155189b77ad4c not found: ID does not exist"
Oct 30 19:37:58.962968 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.962889    5043 scope.go:122] "RemoveContainer" containerID="b1d3d9f4c5634497a9f52cb9018117c2a4b9b81baf656cf5a88532f62f9eafec"
Oct 30 19:37:58.963199 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:37:58.963179    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1d3d9f4c5634497a9f52cb9018117c2a4b9b81baf656cf5a88532f62f9eafec\": container with ID starting with b1d3d9f4c5634497a9f52cb9018117c2a4b9b81baf656cf5a88532f62f9eafec not found: ID does not exist" containerID="b1d3d9f4c5634497a9f52cb9018117c2a4b9b81baf656cf5a88532f62f9eafec"
Oct 30 19:37:58.963245 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:58.963208    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1d3d9f4c5634497a9f52cb9018117c2a4b9b81baf656cf5a88532f62f9eafec"} err="failed to get container status \"b1d3d9f4c5634497a9f52cb9018117c2a4b9b81baf656cf5a88532f62f9eafec\": rpc error: code = NotFound desc = could not find container \"b1d3d9f4c5634497a9f52cb9018117c2a4b9b81baf656cf5a88532f62f9eafec\": container with ID starting with b1d3d9f4c5634497a9f52cb9018117c2a4b9b81baf656cf5a88532f62f9eafec not found: ID does not exist"
Oct 30 19:37:59.276002 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.275965    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56581f7b-wzmj8"]
Oct 30 19:37:59.277143 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.277105    5043 cpu_manager.go:414] "containerMap: removing container" podUID="869b0618-5974-4bc0-91cf-339f483d115f" containerName="revoke-keys"
Oct 30 19:37:59.277143 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.277144    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="869b0618-5974-4bc0-91cf-339f483d115f" containerName="revoke-keys"
Oct 30 19:37:59.277327 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.277181    5043 cpu_manager.go:414] "containerMap: removing container" podUID="dfd35ad6-36ee-44ba-8d74-021c61177b3b" containerName="istio-proxy"
Oct 30 19:37:59.277327 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.277189    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfd35ad6-36ee-44ba-8d74-021c61177b3b" containerName="istio-proxy"
Oct 30 19:37:59.277327 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.277226    5043 cpu_manager.go:414] "containerMap: removing container" podUID="6ae382e3-be21-4e69-bf70-a8c0b8517789" containerName="istio-proxy"
Oct 30 19:37:59.277327 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.277235    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ae382e3-be21-4e69-bf70-a8c0b8517789" containerName="istio-proxy"
Oct 30 19:37:59.277327 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.277254    5043 cpu_manager.go:414] "containerMap: removing container" podUID="dfd35ad6-36ee-44ba-8d74-021c61177b3b" containerName="istio-proxy"
Oct 30 19:37:59.277327 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.277260    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfd35ad6-36ee-44ba-8d74-021c61177b3b" containerName="istio-proxy"
Oct 30 19:37:59.277639 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.277508    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ae382e3-be21-4e69-bf70-a8c0b8517789" containerName="istio-proxy"
Oct 30 19:37:59.277639 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.277526    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfd35ad6-36ee-44ba-8d74-021c61177b3b" containerName="istio-proxy"
Oct 30 19:37:59.277639 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.277552    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="869b0618-5974-4bc0-91cf-339f483d115f" containerName="revoke-keys"
Oct 30 19:37:59.284645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.284191    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56581f7b-wzmj8"
Oct 30 19:37:59.292754 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.292510    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56581f7b-wzmj8"]
Oct 30 19:37:59.309032 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.307967    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:37:59.309032 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.308010    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:37:59.309032 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.308057    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:37:59.309032 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.308120    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:37:59.445097 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.445013    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56581f7b-wzmj8\" (UID: \"6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56581f7b-wzmj8"
Oct 30 19:37:59.445278 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.445183    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxbws\" (UniqueName: \"kubernetes.io/projected/6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162-kube-api-access-kxbws\") pod \"maas-api-revoke-sub-56581f7b-wzmj8\" (UID: \"6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56581f7b-wzmj8"
Oct 30 19:37:59.445396 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.445361    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56581f7b-wzmj8\" (UID: \"6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56581f7b-wzmj8"
Oct 30 19:37:59.547695 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.547659    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56581f7b-wzmj8\" (UID: \"6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56581f7b-wzmj8"
Oct 30 19:37:59.547899 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.547786    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxbws\" (UniqueName: \"kubernetes.io/projected/6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162-kube-api-access-kxbws\") pod \"maas-api-revoke-sub-56581f7b-wzmj8\" (UID: \"6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56581f7b-wzmj8"
Oct 30 19:37:59.547899 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.547886    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56581f7b-wzmj8\" (UID: \"6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56581f7b-wzmj8"
Oct 30 19:37:59.548540 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.548519    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56581f7b-wzmj8\" (UID: \"6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56581f7b-wzmj8"
Oct 30 19:37:59.550352 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.550328    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56581f7b-wzmj8\" (UID: \"6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56581f7b-wzmj8"
Oct 30 19:37:59.556570 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.556540    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxbws\" (UniqueName: \"kubernetes.io/projected/6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162-kube-api-access-kxbws\") pod \"maas-api-revoke-sub-56581f7b-wzmj8\" (UID: \"6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56581f7b-wzmj8"
Oct 30 19:37:59.610555 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.610523    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56581f7b-wzmj8"
Oct 30 19:37:59.841318 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.841283    5043 generic.go:356] "Generic (PLEG): container finished" podID="e8f2440f-1238-46f6-ba78-90f9c38b574d" containerID="6aa0c38ba63755a080259cb76d2111e35ee8be8bbf1da42d55784ed8b17ac664" exitCode=0
Oct 30 19:37:59.841483 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.841357    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9" event={"ID":"e8f2440f-1238-46f6-ba78-90f9c38b574d","Type":"ContainerDied","Data":"6aa0c38ba63755a080259cb76d2111e35ee8be8bbf1da42d55784ed8b17ac664"}
Oct 30 19:37:59.853447 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.853422    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9"
Oct 30 19:37:59.907327 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:37:59.907281    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a87ff2b_ff7d_4e7c_b2ca_f1c78cc4f162.slice/crio-a76e67a9c28670d5d79e9a693ce8bc41b987bac7216fcb5345d32666c0717d78 WatchSource:0}: Error finding container a76e67a9c28670d5d79e9a693ce8bc41b987bac7216fcb5345d32666c0717d78: Status 404 returned error can't find the container with id a76e67a9c28670d5d79e9a693ce8bc41b987bac7216fcb5345d32666c0717d78
Oct 30 19:37:59.910360 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.910336    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56581f7b-wzmj8"]
Oct 30 19:37:59.955662 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.955635    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8f2440f-1238-46f6-ba78-90f9c38b574d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8f2440f-1238-46f6-ba78-90f9c38b574d-istio-data\") pod \"e8f2440f-1238-46f6-ba78-90f9c38b574d\" (UID: \"e8f2440f-1238-46f6-ba78-90f9c38b574d\") "
Oct 30 19:37:59.955830 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.955680    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8f2440f-1238-46f6-ba78-90f9c38b574d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8f2440f-1238-46f6-ba78-90f9c38b574d-workload-certs\") pod \"e8f2440f-1238-46f6-ba78-90f9c38b574d\" (UID: \"e8f2440f-1238-46f6-ba78-90f9c38b574d\") "
Oct 30 19:37:59.955830 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.955719    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8f2440f-1238-46f6-ba78-90f9c38b574d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8f2440f-1238-46f6-ba78-90f9c38b574d-istiod-ca-cert\") pod \"e8f2440f-1238-46f6-ba78-90f9c38b574d\" (UID: \"e8f2440f-1238-46f6-ba78-90f9c38b574d\") "
Oct 30 19:37:59.955830 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.955764    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8f2440f-1238-46f6-ba78-90f9c38b574d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8f2440f-1238-46f6-ba78-90f9c38b574d-istio-envoy\") pod \"e8f2440f-1238-46f6-ba78-90f9c38b574d\" (UID: \"e8f2440f-1238-46f6-ba78-90f9c38b574d\") "
Oct 30 19:37:59.955830 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.955780    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8f2440f-1238-46f6-ba78-90f9c38b574d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8f2440f-1238-46f6-ba78-90f9c38b574d-credential-socket\") pod \"e8f2440f-1238-46f6-ba78-90f9c38b574d\" (UID: \"e8f2440f-1238-46f6-ba78-90f9c38b574d\") "
Oct 30 19:37:59.955830 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.955817    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8f2440f-1238-46f6-ba78-90f9c38b574d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8f2440f-1238-46f6-ba78-90f9c38b574d-istio-podinfo\") pod \"e8f2440f-1238-46f6-ba78-90f9c38b574d\" (UID: \"e8f2440f-1238-46f6-ba78-90f9c38b574d\") "
Oct 30 19:37:59.956072 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.955850    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8f2440f-1238-46f6-ba78-90f9c38b574d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8f2440f-1238-46f6-ba78-90f9c38b574d-workload-socket\") pod \"e8f2440f-1238-46f6-ba78-90f9c38b574d\" (UID: \"e8f2440f-1238-46f6-ba78-90f9c38b574d\") "
Oct 30 19:37:59.956072 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.955890    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8f2440f-1238-46f6-ba78-90f9c38b574d-kube-api-access-n6w8l\" (UniqueName: \"kubernetes.io/projected/e8f2440f-1238-46f6-ba78-90f9c38b574d-kube-api-access-n6w8l\") pod \"e8f2440f-1238-46f6-ba78-90f9c38b574d\" (UID: \"e8f2440f-1238-46f6-ba78-90f9c38b574d\") "
Oct 30 19:37:59.956163 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.956155    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8f2440f-1238-46f6-ba78-90f9c38b574d-istio-token\" (UniqueName: \"kubernetes.io/projected/e8f2440f-1238-46f6-ba78-90f9c38b574d-istio-token\") pod \"e8f2440f-1238-46f6-ba78-90f9c38b574d\" (UID: \"e8f2440f-1238-46f6-ba78-90f9c38b574d\") "
Oct 30 19:37:59.956680 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.956151    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8f2440f-1238-46f6-ba78-90f9c38b574d-istiod-ca-cert" pod "e8f2440f-1238-46f6-ba78-90f9c38b574d" (UID: "e8f2440f-1238-46f6-ba78-90f9c38b574d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:37:59.957301 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.956185    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8f2440f-1238-46f6-ba78-90f9c38b574d-workload-socket" pod "e8f2440f-1238-46f6-ba78-90f9c38b574d" (UID: "e8f2440f-1238-46f6-ba78-90f9c38b574d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:59.957301 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.956647    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8f2440f-1238-46f6-ba78-90f9c38b574d-workload-certs" pod "e8f2440f-1238-46f6-ba78-90f9c38b574d" (UID: "e8f2440f-1238-46f6-ba78-90f9c38b574d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:59.957712 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.957690    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8f2440f-1238-46f6-ba78-90f9c38b574d-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:59.958128 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.958094    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8f2440f-1238-46f6-ba78-90f9c38b574d-credential-socket" pod "e8f2440f-1238-46f6-ba78-90f9c38b574d" (UID: "e8f2440f-1238-46f6-ba78-90f9c38b574d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:59.958412 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.958394    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8f2440f-1238-46f6-ba78-90f9c38b574d-istio-data" pod "e8f2440f-1238-46f6-ba78-90f9c38b574d" (UID: "e8f2440f-1238-46f6-ba78-90f9c38b574d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:59.958949 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.958918    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8f2440f-1238-46f6-ba78-90f9c38b574d-istio-token" pod "e8f2440f-1238-46f6-ba78-90f9c38b574d" (UID: "e8f2440f-1238-46f6-ba78-90f9c38b574d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:59.959108 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.959083    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8f2440f-1238-46f6-ba78-90f9c38b574d-kube-api-access-n6w8l" pod "e8f2440f-1238-46f6-ba78-90f9c38b574d" (UID: "e8f2440f-1238-46f6-ba78-90f9c38b574d"). InnerVolumeSpecName "kube-api-access-n6w8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:59.960079 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.960054    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8f2440f-1238-46f6-ba78-90f9c38b574d-istio-podinfo" pod "e8f2440f-1238-46f6-ba78-90f9c38b574d" (UID: "e8f2440f-1238-46f6-ba78-90f9c38b574d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:37:59.960192 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:37:59.960170    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8f2440f-1238-46f6-ba78-90f9c38b574d-istio-envoy" pod "e8f2440f-1238-46f6-ba78-90f9c38b574d" (UID: "e8f2440f-1238-46f6-ba78-90f9c38b574d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:00.060122 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:00.060010    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8f2440f-1238-46f6-ba78-90f9c38b574d-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:00.060122 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:00.060046    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8f2440f-1238-46f6-ba78-90f9c38b574d-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:00.060122 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:00.060058    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8f2440f-1238-46f6-ba78-90f9c38b574d-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:00.060122 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:00.060067    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8f2440f-1238-46f6-ba78-90f9c38b574d-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:00.060122 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:00.060076    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8f2440f-1238-46f6-ba78-90f9c38b574d-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:00.060122 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:00.060084    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8f2440f-1238-46f6-ba78-90f9c38b574d-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:00.060122 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:00.060094    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8f2440f-1238-46f6-ba78-90f9c38b574d-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:00.060122 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:00.060102    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6w8l\" (UniqueName: \"kubernetes.io/projected/e8f2440f-1238-46f6-ba78-90f9c38b574d-kube-api-access-n6w8l\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:00.652814 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:00.652782    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ae382e3-be21-4e69-bf70-a8c0b8517789" path="/var/lib/kubelet/pods/6ae382e3-be21-4e69-bf70-a8c0b8517789/volumes"
Oct 30 19:38:00.653434 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:00.653417    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfd35ad6-36ee-44ba-8d74-021c61177b3b" path="/var/lib/kubelet/pods/dfd35ad6-36ee-44ba-8d74-021c61177b3b/volumes"
Oct 30 19:38:00.854265 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:00.854166    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56581f7b-wzmj8" event={"ID":"6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162","Type":"ContainerStarted","Data":"c51b6a9d96b1f22f01da463d572792872c36f3b81a96cc893d6e22c43b955085"}
Oct 30 19:38:00.854265 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:00.854232    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56581f7b-wzmj8" event={"ID":"6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162","Type":"ContainerStarted","Data":"a76e67a9c28670d5d79e9a693ce8bc41b987bac7216fcb5345d32666c0717d78"}
Oct 30 19:38:00.855723 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:00.855703    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9"
Oct 30 19:38:00.855823 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:00.855770    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9" event={"ID":"e8f2440f-1238-46f6-ba78-90f9c38b574d","Type":"ContainerDied","Data":"00036b6651de3a17527070a9d27088be699730eb3f5042b7ecb4448acd91d08d"}
Oct 30 19:38:00.855895 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:00.855835    5043 scope.go:122] "RemoveContainer" containerID="6aa0c38ba63755a080259cb76d2111e35ee8be8bbf1da42d55784ed8b17ac664"
Oct 30 19:38:00.880894 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:00.880834    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56581f7b-wzmj8" podStartSLOduration=1.880818377 podStartE2EDuration="1.880818377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:00.878616361 +0000 UTC m=+1300.389676328" watchObservedRunningTime="2026-10-30 19:38:00.880818377 +0000 UTC m=+1300.391878362"
Oct 30 19:38:00.904648 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:00.904611    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9"]
Oct 30 19:38:00.909804 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:00.909775    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-jx2g9"]
Oct 30 19:38:01.846722 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:01.846685    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d1d256-5b889bbcc5-qb7k8"]
Oct 30 19:38:01.847751 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:01.847701    5043 cpu_manager.go:414] "containerMap: removing container" podUID="e8f2440f-1238-46f6-ba78-90f9c38b574d" containerName="istio-proxy"
Oct 30 19:38:01.847751 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:01.847749    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8f2440f-1238-46f6-ba78-90f9c38b574d" containerName="istio-proxy"
Oct 30 19:38:01.847926 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:01.847870    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8f2440f-1238-46f6-ba78-90f9c38b574d" containerName="istio-proxy"
Oct 30 19:38:01.847926 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:01.847902    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfd35ad6-36ee-44ba-8d74-021c61177b3b" containerName="istio-proxy"
Oct 30 19:38:01.851987 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:01.851970    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d1d256-5b889bbcc5-qb7k8"
Oct 30 19:38:01.864996 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:01.864933    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-d1d256\""
Oct 30 19:38:01.880268 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:01.880227    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d1d256-5b889bbcc5-qb7k8"]
Oct 30 19:38:01.892953 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:01.892925    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d1d256-5b65b66d9d-2djrn"]
Oct 30 19:38:01.893297 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:01.893273    5043 generic.go:356] "Generic (PLEG): container finished" podID="6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162" containerID="c51b6a9d96b1f22f01da463d572792872c36f3b81a96cc893d6e22c43b955085" exitCode=0
Oct 30 19:38:01.899314 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:01.899278    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56581f7b-wzmj8" event={"ID":"6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162","Type":"ContainerDied","Data":"c51b6a9d96b1f22f01da463d572792872c36f3b81a96cc893d6e22c43b955085"}
Oct 30 19:38:01.899456 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:01.899389    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d1d256-5b65b66d9d-2djrn"
Oct 30 19:38:01.899858 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:01.899836    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-d1d256-dockercfg-skh8w\""
Oct 30 19:38:01.912654 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:01.912608    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d1d256-5b65b66d9d-2djrn"]
Oct 30 19:38:02.174865 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:02.132791    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6m8t\" (UniqueName: \"kubernetes.io/projected/2f72b377-925e-4664-8b40-1db41ac5fa15-kube-api-access-f6m8t\") pod \"payload-processing-e2e-api-a-d1d256-5b65b66d9d-2djrn\" (UID: \"2f72b377-925e-4664-8b40-1db41ac5fa15\") " pod="openshift-ingress/payload-processing-e2e-api-a-d1d256-5b65b66d9d-2djrn"
Oct 30 19:38:02.174865 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:02.132949    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fe8aff5-f319-4891-9108-274458478a6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d1d256-5b889bbcc5-qb7k8\" (UID: \"6fe8aff5-f319-4891-9108-274458478a6e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d1d256-5b889bbcc5-qb7k8"
Oct 30 19:38:02.174865 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:02.133184    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f72b377-925e-4664-8b40-1db41ac5fa15-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d1d256-5b65b66d9d-2djrn\" (UID: \"2f72b377-925e-4664-8b40-1db41ac5fa15\") " pod="openshift-ingress/payload-processing-e2e-api-a-d1d256-5b65b66d9d-2djrn"
Oct 30 19:38:02.174865 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:02.133320    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hxsd\" (UniqueName: \"kubernetes.io/projected/6fe8aff5-f319-4891-9108-274458478a6e-kube-api-access-2hxsd\") pod \"payload-pre-processing-e2e-api-a-d1d256-5b889bbcc5-qb7k8\" (UID: \"6fe8aff5-f319-4891-9108-274458478a6e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d1d256-5b889bbcc5-qb7k8"
Oct 30 19:38:02.237804 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:02.237771    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6m8t\" (UniqueName: \"kubernetes.io/projected/2f72b377-925e-4664-8b40-1db41ac5fa15-kube-api-access-f6m8t\") pod \"payload-processing-e2e-api-a-d1d256-5b65b66d9d-2djrn\" (UID: \"2f72b377-925e-4664-8b40-1db41ac5fa15\") " pod="openshift-ingress/payload-processing-e2e-api-a-d1d256-5b65b66d9d-2djrn"
Oct 30 19:38:02.238095 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:02.238076    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fe8aff5-f319-4891-9108-274458478a6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d1d256-5b889bbcc5-qb7k8\" (UID: \"6fe8aff5-f319-4891-9108-274458478a6e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d1d256-5b889bbcc5-qb7k8"
Oct 30 19:38:02.238359 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:02.238345    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f72b377-925e-4664-8b40-1db41ac5fa15-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d1d256-5b65b66d9d-2djrn\" (UID: \"2f72b377-925e-4664-8b40-1db41ac5fa15\") " pod="openshift-ingress/payload-processing-e2e-api-a-d1d256-5b65b66d9d-2djrn"
Oct 30 19:38:02.238622 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:02.238555    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hxsd\" (UniqueName: \"kubernetes.io/projected/6fe8aff5-f319-4891-9108-274458478a6e-kube-api-access-2hxsd\") pod \"payload-pre-processing-e2e-api-a-d1d256-5b889bbcc5-qb7k8\" (UID: \"6fe8aff5-f319-4891-9108-274458478a6e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d1d256-5b889bbcc5-qb7k8"
Oct 30 19:38:02.254531 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:02.254498    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fe8aff5-f319-4891-9108-274458478a6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d1d256-5b889bbcc5-qb7k8\" (UID: \"6fe8aff5-f319-4891-9108-274458478a6e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d1d256-5b889bbcc5-qb7k8"
Oct 30 19:38:02.255578 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:02.255552    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f72b377-925e-4664-8b40-1db41ac5fa15-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d1d256-5b65b66d9d-2djrn\" (UID: \"2f72b377-925e-4664-8b40-1db41ac5fa15\") " pod="openshift-ingress/payload-processing-e2e-api-a-d1d256-5b65b66d9d-2djrn"
Oct 30 19:38:02.324837 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:02.324055    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hxsd\" (UniqueName: \"kubernetes.io/projected/6fe8aff5-f319-4891-9108-274458478a6e-kube-api-access-2hxsd\") pod \"payload-pre-processing-e2e-api-a-d1d256-5b889bbcc5-qb7k8\" (UID: \"6fe8aff5-f319-4891-9108-274458478a6e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d1d256-5b889bbcc5-qb7k8"
Oct 30 19:38:02.329441 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:02.327823    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6m8t\" (UniqueName: \"kubernetes.io/projected/2f72b377-925e-4664-8b40-1db41ac5fa15-kube-api-access-f6m8t\") pod \"payload-processing-e2e-api-a-d1d256-5b65b66d9d-2djrn\" (UID: \"2f72b377-925e-4664-8b40-1db41ac5fa15\") " pod="openshift-ingress/payload-processing-e2e-api-a-d1d256-5b65b66d9d-2djrn"
Oct 30 19:38:02.487985 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:02.487949    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d1d256-5b889bbcc5-qb7k8"
Oct 30 19:38:02.521812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:02.520877    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d1d256-5b65b66d9d-2djrn"
Oct 30 19:38:02.666243 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:02.665163    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8f2440f-1238-46f6-ba78-90f9c38b574d" path="/var/lib/kubelet/pods/e8f2440f-1238-46f6-ba78-90f9c38b574d/volumes"
Oct 30 19:38:02.781096 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:02.781005    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6a6348-677c897664-5q8vf"]
Oct 30 19:38:02.813227 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:02.813200    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6a6348-79bcfb7655-cpxz9"]
Oct 30 19:38:02.821428 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:02.821380    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d1d256-5b889bbcc5-qb7k8"]
Oct 30 19:38:02.862692 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:02.862597    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d1d256-5b65b66d9d-2djrn"]
Oct 30 19:38:02.877034 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:38:02.875474    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f72b377_925e_4664_8b40_1db41ac5fa15.slice/crio-99b62b9ebbcf88ff12add130e083fb03a1af25a13908baf21e951d69f4d62548 WatchSource:0}: Error finding container 99b62b9ebbcf88ff12add130e083fb03a1af25a13908baf21e951d69f4d62548: Status 404 returned error can't find the container with id 99b62b9ebbcf88ff12add130e083fb03a1af25a13908baf21e951d69f4d62548
Oct 30 19:38:02.915523 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:02.914207    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d1d256-5b65b66d9d-2djrn" event={"ID":"2f72b377-925e-4664-8b40-1db41ac5fa15","Type":"ContainerStarted","Data":"99b62b9ebbcf88ff12add130e083fb03a1af25a13908baf21e951d69f4d62548"}
Oct 30 19:38:02.915908 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:02.915877    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d1d256-5b889bbcc5-qb7k8" event={"ID":"6fe8aff5-f319-4891-9108-274458478a6e","Type":"ContainerStarted","Data":"61b17590a7c535476b39208e70efcd26d85eb7bfe4ca5b3326588f3a7b59a14a"}
Oct 30 19:38:03.094942 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:03.094914    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56581f7b-wzmj8"
Oct 30 19:38:03.159223 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:03.156087    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162-maas-api-openshift-ca\") pod \"6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162\" (UID: \"6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162\") "
Oct 30 19:38:03.159223 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:03.156329    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162-maas-api-xks-ca\") pod \"6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162\" (UID: \"6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162\") "
Oct 30 19:38:03.159223 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:03.156411    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162-kube-api-access-kxbws\" (UniqueName: \"kubernetes.io/projected/6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162-kube-api-access-kxbws\") pod \"6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162\" (UID: \"6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162\") "
Oct 30 19:38:03.168775 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:03.159943    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162-maas-api-openshift-ca" pod "6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162" (UID: "6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:38:03.317765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:03.310341    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:03.386757 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:03.383029    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162-maas-api-xks-ca" pod "6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162" (UID: "6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:38:03.394210 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:03.394169    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162-kube-api-access-kxbws" pod "6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162" (UID: "6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162"). InnerVolumeSpecName "kube-api-access-kxbws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:03.420793 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:03.420759    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:03.421052 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:03.421033    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxbws\" (UniqueName: \"kubernetes.io/projected/6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162-kube-api-access-kxbws\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:03.473760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:03.467533    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-60577a-fg75m"]
Oct 30 19:38:03.480634 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:03.475884    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-60577a-fg75m"]
Oct 30 19:38:03.930178 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:03.930144    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d1d256-5b889bbcc5-qb7k8" event={"ID":"6fe8aff5-f319-4891-9108-274458478a6e","Type":"ContainerStarted","Data":"df10efb295e320fafbde8f149f56e7a2544b28d181def23037ccb3a49c9bc0cb"}
Oct 30 19:38:03.930621 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:03.930232    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d1d256-5b889bbcc5-qb7k8"
Oct 30 19:38:03.932229 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:03.932208    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56581f7b-wzmj8"
Oct 30 19:38:03.932376 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:03.932236    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56581f7b-wzmj8" event={"ID":"6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162","Type":"ContainerDied","Data":"a76e67a9c28670d5d79e9a693ce8bc41b987bac7216fcb5345d32666c0717d78"}
Oct 30 19:38:03.932376 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:03.932286    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a76e67a9c28670d5d79e9a693ce8bc41b987bac7216fcb5345d32666c0717d78"
Oct 30 19:38:03.934270 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:03.934238    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d1d256-5b65b66d9d-2djrn" event={"ID":"2f72b377-925e-4664-8b40-1db41ac5fa15","Type":"ContainerStarted","Data":"4f8ab4fb2c02061ec8e432751dfbaa4199ffbc7876a6d87e899c7b6cf3aace03"}
Oct 30 19:38:03.934387 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:03.934342    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-d1d256-5b65b66d9d-2djrn"
Oct 30 19:38:03.979729 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:03.979671    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d1d256-5b889bbcc5-qb7k8" podStartSLOduration=2.979657581 podStartE2EDuration="2.979657581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:03.97217936 +0000 UTC m=+1303.483239326" watchObservedRunningTime="2026-10-30 19:38:03.979657581 +0000 UTC m=+1303.490717545"
Oct 30 19:38:04.008892 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:04.008831    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-d1d256-5b65b66d9d-2djrn" podStartSLOduration=3.008808191 podStartE2EDuration="3.008808191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:04.007685935 +0000 UTC m=+1303.518745897" watchObservedRunningTime="2026-10-30 19:38:04.008808191 +0000 UTC m=+1303.519868161"
Oct 30 19:38:04.780768 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:04.780581    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88bba8ee-11c4-48f7-80c8-d3a636ea8a1c" path="/var/lib/kubelet/pods/88bba8ee-11c4-48f7-80c8-d3a636ea8a1c/volumes"
Oct 30 19:38:07.781826 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:07.781666    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-6a6348-677c897664-5q8vf" podUID="59510403-878b-4f26-895c-cf3d22ef6ece" containerName="payload-processing" containerID="cri-o://11b1e0939f27f6704e7c2d53a6ffd0514e86d6e051d12707a9f417047c854afb" gracePeriod=25
Oct 30 19:38:07.814084 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:07.814025    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a6348-79bcfb7655-cpxz9" podUID="0ff1108a-87ef-44f8-a85b-42e48bf99de9" containerName="payload-pre-processing" containerID="cri-o://d6cbddd51e225a80e22333dcfd89dd23bdd21b5e654b43ca79065ec79da385e3" gracePeriod=25
Oct 30 19:38:08.009662 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:08.009440    5043 generic.go:356] "Generic (PLEG): container finished" podID="59510403-878b-4f26-895c-cf3d22ef6ece" containerID="11b1e0939f27f6704e7c2d53a6ffd0514e86d6e051d12707a9f417047c854afb" exitCode=0
Oct 30 19:38:08.009662 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:08.009572    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6a6348-677c897664-5q8vf" event={"ID":"59510403-878b-4f26-895c-cf3d22ef6ece","Type":"ContainerDied","Data":"11b1e0939f27f6704e7c2d53a6ffd0514e86d6e051d12707a9f417047c854afb"}
Oct 30 19:38:08.012202 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:08.012166    5043 generic.go:356] "Generic (PLEG): container finished" podID="0ff1108a-87ef-44f8-a85b-42e48bf99de9" containerID="d6cbddd51e225a80e22333dcfd89dd23bdd21b5e654b43ca79065ec79da385e3" exitCode=0
Oct 30 19:38:08.012516 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:08.012498    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a6348-79bcfb7655-cpxz9" event={"ID":"0ff1108a-87ef-44f8-a85b-42e48bf99de9","Type":"ContainerDied","Data":"d6cbddd51e225a80e22333dcfd89dd23bdd21b5e654b43ca79065ec79da385e3"}
Oct 30 19:38:08.234316 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:08.234281    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a6348-79bcfb7655-cpxz9"
Oct 30 19:38:08.324479 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:08.324451    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6a6348-677c897664-5q8vf"
Oct 30 19:38:08.352340 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:08.352307    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ff1108a-87ef-44f8-a85b-42e48bf99de9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ff1108a-87ef-44f8-a85b-42e48bf99de9-plugins-config-volume\") pod \"0ff1108a-87ef-44f8-a85b-42e48bf99de9\" (UID: \"0ff1108a-87ef-44f8-a85b-42e48bf99de9\") "
Oct 30 19:38:08.352671 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:08.352652    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ff1108a-87ef-44f8-a85b-42e48bf99de9-kube-api-access-s59n6\" (UniqueName: \"kubernetes.io/projected/0ff1108a-87ef-44f8-a85b-42e48bf99de9-kube-api-access-s59n6\") pod \"0ff1108a-87ef-44f8-a85b-42e48bf99de9\" (UID: \"0ff1108a-87ef-44f8-a85b-42e48bf99de9\") "
Oct 30 19:38:08.352779 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:08.352749    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ff1108a-87ef-44f8-a85b-42e48bf99de9-plugins-config-volume" pod "0ff1108a-87ef-44f8-a85b-42e48bf99de9" (UID: "0ff1108a-87ef-44f8-a85b-42e48bf99de9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:38:08.353295 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:08.353273    5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ff1108a-87ef-44f8-a85b-42e48bf99de9-plugins-config-volume\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:08.355058 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:08.355028    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ff1108a-87ef-44f8-a85b-42e48bf99de9-kube-api-access-s59n6" pod "0ff1108a-87ef-44f8-a85b-42e48bf99de9" (UID: "0ff1108a-87ef-44f8-a85b-42e48bf99de9"). InnerVolumeSpecName "kube-api-access-s59n6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:08.454294 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:08.454264    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59510403-878b-4f26-895c-cf3d22ef6ece-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59510403-878b-4f26-895c-cf3d22ef6ece-plugins-config-volume\") pod \"59510403-878b-4f26-895c-cf3d22ef6ece\" (UID: \"59510403-878b-4f26-895c-cf3d22ef6ece\") "
Oct 30 19:38:08.454491 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:08.454314    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59510403-878b-4f26-895c-cf3d22ef6ece-kube-api-access-jrfvv\" (UniqueName: \"kubernetes.io/projected/59510403-878b-4f26-895c-cf3d22ef6ece-kube-api-access-jrfvv\") pod \"59510403-878b-4f26-895c-cf3d22ef6ece\" (UID: \"59510403-878b-4f26-895c-cf3d22ef6ece\") "
Oct 30 19:38:08.454613 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:08.454588    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59510403-878b-4f26-895c-cf3d22ef6ece-plugins-config-volume" pod "59510403-878b-4f26-895c-cf3d22ef6ece" (UID: "59510403-878b-4f26-895c-cf3d22ef6ece"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:38:08.454886 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:08.454870    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s59n6\" (UniqueName: \"kubernetes.io/projected/0ff1108a-87ef-44f8-a85b-42e48bf99de9-kube-api-access-s59n6\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:08.454930 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:08.454890    5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59510403-878b-4f26-895c-cf3d22ef6ece-plugins-config-volume\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:08.456806 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:08.456778    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59510403-878b-4f26-895c-cf3d22ef6ece-kube-api-access-jrfvv" pod "59510403-878b-4f26-895c-cf3d22ef6ece" (UID: "59510403-878b-4f26-895c-cf3d22ef6ece"). InnerVolumeSpecName "kube-api-access-jrfvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:08.556633 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:08.556539    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrfvv\" (UniqueName: \"kubernetes.io/projected/59510403-878b-4f26-895c-cf3d22ef6ece-kube-api-access-jrfvv\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:08.852635 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:08.850910    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-156838-6749758ddc-tnv4d"
Oct 30 19:38:08.855577 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:08.855549    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-156838-79f475dcd-x2qg9"
Oct 30 19:38:09.029168 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:09.029143    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6a6348-677c897664-5q8vf"
Oct 30 19:38:09.029392 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:09.029170    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6a6348-677c897664-5q8vf" event={"ID":"59510403-878b-4f26-895c-cf3d22ef6ece","Type":"ContainerDied","Data":"2ff02125202640b796e77e870c07a75f289c314e03fa6a71d7fa7afc408b2ab6"}
Oct 30 19:38:09.029392 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:09.029243    5043 scope.go:122] "RemoveContainer" containerID="11b1e0939f27f6704e7c2d53a6ffd0514e86d6e051d12707a9f417047c854afb"
Oct 30 19:38:09.031553 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:09.031519    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a6348-79bcfb7655-cpxz9" event={"ID":"0ff1108a-87ef-44f8-a85b-42e48bf99de9","Type":"ContainerDied","Data":"574cf2e65ffce7d2eed60fa9f63046c337236a287cb662df6bf2ffd57fdc7b11"}
Oct 30 19:38:09.038556 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:09.038495    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6a6348-79bcfb7655-cpxz9"
Oct 30 19:38:09.050071 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:09.050049    5043 scope.go:122] "RemoveContainer" containerID="d6cbddd51e225a80e22333dcfd89dd23bdd21b5e654b43ca79065ec79da385e3"
Oct 30 19:38:09.063903 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:09.063870    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6a6348-79bcfb7655-cpxz9"]
Oct 30 19:38:09.069873 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:09.069845    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6a6348-79bcfb7655-cpxz9"]
Oct 30 19:38:09.080989 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:09.079445    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6a6348-677c897664-5q8vf"]
Oct 30 19:38:09.085109 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:09.085079    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6a6348-677c897664-5q8vf"]
Oct 30 19:38:09.532802 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:09.532762    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-395ccc-zmhsj"]
Oct 30 19:38:09.533570 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:09.533554    5043 cpu_manager.go:414] "containerMap: removing container" podUID="0ff1108a-87ef-44f8-a85b-42e48bf99de9" containerName="payload-pre-processing"
Oct 30 19:38:09.533621 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:09.533572    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ff1108a-87ef-44f8-a85b-42e48bf99de9" containerName="payload-pre-processing"
Oct 30 19:38:09.533621 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:09.533587    5043 cpu_manager.go:414] "containerMap: removing container" podUID="59510403-878b-4f26-895c-cf3d22ef6ece" containerName="payload-processing"
Oct 30 19:38:09.533621 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:09.533593    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59510403-878b-4f26-895c-cf3d22ef6ece" containerName="payload-processing"
Oct 30 19:38:09.533621 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:09.533607    5043 cpu_manager.go:414] "containerMap: removing container" podUID="6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162" containerName="revoke-keys"
Oct 30 19:38:09.533621 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:09.533612    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162" containerName="revoke-keys"
Oct 30 19:38:09.533818 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:09.533780    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="59510403-878b-4f26-895c-cf3d22ef6ece" containerName="payload-processing"
Oct 30 19:38:09.533818 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:09.533792    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162" containerName="revoke-keys"
Oct 30 19:38:09.533818 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:09.533812    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ff1108a-87ef-44f8-a85b-42e48bf99de9" containerName="payload-pre-processing"
Oct 30 19:38:09.539054 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:09.539032    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-395ccc-zmhsj"
Oct 30 19:38:09.543521 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:09.543488    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:38:09.543654 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:09.543542    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:38:09.545035 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:09.544520    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:38:09.545035 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:09.544533    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:38:09.549184 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:09.549161    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-395ccc-zmhsj"]
Oct 30 19:38:09.690713 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:09.690668    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be1be942-2abe-4c86-829b-474a51943d48-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-395ccc-zmhsj\" (UID: \"be1be942-2abe-4c86-829b-474a51943d48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-395ccc-zmhsj"
Oct 30 19:38:09.690713 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:09.690715    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltklq\" (UniqueName: \"kubernetes.io/projected/be1be942-2abe-4c86-829b-474a51943d48-kube-api-access-ltklq\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-395ccc-zmhsj\" (UID: \"be1be942-2abe-4c86-829b-474a51943d48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-395ccc-zmhsj"
Oct 30 19:38:09.691112 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:09.691095    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be1be942-2abe-4c86-829b-474a51943d48-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-395ccc-zmhsj\" (UID: \"be1be942-2abe-4c86-829b-474a51943d48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-395ccc-zmhsj"
Oct 30 19:38:09.793226 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:09.793130    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be1be942-2abe-4c86-829b-474a51943d48-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-395ccc-zmhsj\" (UID: \"be1be942-2abe-4c86-829b-474a51943d48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-395ccc-zmhsj"
Oct 30 19:38:09.793404 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:09.793230    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be1be942-2abe-4c86-829b-474a51943d48-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-395ccc-zmhsj\" (UID: \"be1be942-2abe-4c86-829b-474a51943d48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-395ccc-zmhsj"
Oct 30 19:38:09.793404 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:09.793262    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltklq\" (UniqueName: \"kubernetes.io/projected/be1be942-2abe-4c86-829b-474a51943d48-kube-api-access-ltklq\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-395ccc-zmhsj\" (UID: \"be1be942-2abe-4c86-829b-474a51943d48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-395ccc-zmhsj"
Oct 30 19:38:09.794039 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:09.794013    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be1be942-2abe-4c86-829b-474a51943d48-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-395ccc-zmhsj\" (UID: \"be1be942-2abe-4c86-829b-474a51943d48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-395ccc-zmhsj"
Oct 30 19:38:09.796093 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:09.796068    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be1be942-2abe-4c86-829b-474a51943d48-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-395ccc-zmhsj\" (UID: \"be1be942-2abe-4c86-829b-474a51943d48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-395ccc-zmhsj"
Oct 30 19:38:09.802170 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:09.802137    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltklq\" (UniqueName: \"kubernetes.io/projected/be1be942-2abe-4c86-829b-474a51943d48-kube-api-access-ltklq\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-395ccc-zmhsj\" (UID: \"be1be942-2abe-4c86-829b-474a51943d48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-395ccc-zmhsj"
Oct 30 19:38:09.885413 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:09.885376    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-395ccc-zmhsj"
Oct 30 19:38:10.178005 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:10.177976    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-395ccc-zmhsj"]
Oct 30 19:38:10.649803 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:10.649771    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ff1108a-87ef-44f8-a85b-42e48bf99de9" path="/var/lib/kubelet/pods/0ff1108a-87ef-44f8-a85b-42e48bf99de9/volumes"
Oct 30 19:38:10.650264 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:10.650249    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59510403-878b-4f26-895c-cf3d22ef6ece" path="/var/lib/kubelet/pods/59510403-878b-4f26-895c-cf3d22ef6ece/volumes"
Oct 30 19:38:11.078212 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:11.078182    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-395ccc-zmhsj" event={"ID":"be1be942-2abe-4c86-829b-474a51943d48","Type":"ContainerStarted","Data":"f9ce5733f90ab37dd6011846ca9c8dd59626e8d4887ad81739d2db16f7140570"}
Oct 30 19:38:11.078688 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:11.078240    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-395ccc-zmhsj" event={"ID":"be1be942-2abe-4c86-829b-474a51943d48","Type":"ContainerStarted","Data":"38d5be3b886b4b89ce841b7afa693c4839bfe834c10a86631d934be9227a11b4"}
Oct 30 19:38:11.107759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:11.102019    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-395ccc-zmhsj" podStartSLOduration=2.102000536 podStartE2EDuration="2.102000536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:11.095197347 +0000 UTC m=+1310.606257316" watchObservedRunningTime="2026-10-30 19:38:11.102000536 +0000 UTC m=+1310.613060502"
Oct 30 19:38:11.616729 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:11.616687    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf"]
Oct 30 19:38:11.617038 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:11.616985    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf" podUID="a62f36bd-5442-4679-8616-5b75487cf90a" containerName="istio-proxy" containerID="cri-o://86e91c900aa0277e244ae2e463d59315e17fe21ca62e63c4573764dcbec73794" gracePeriod=30
Oct 30 19:38:11.625155 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:11.625126    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr"]
Oct 30 19:38:11.625361 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:11.625339    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr" podUID="6251d2e1-9d28-4bb8-a667-f415d0f79df0" containerName="istio-proxy" containerID="cri-o://c77d9dd0808935fc5a408dd9c5be241e32f2bdb681f4e19ccf52a5775b214346" gracePeriod=30
Oct 30 19:38:12.090791 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:12.090201    5043 generic.go:356] "Generic (PLEG): container finished" podID="be1be942-2abe-4c86-829b-474a51943d48" containerID="f9ce5733f90ab37dd6011846ca9c8dd59626e8d4887ad81739d2db16f7140570" exitCode=0
Oct 30 19:38:12.090791 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:12.090282    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-395ccc-zmhsj" event={"ID":"be1be942-2abe-4c86-829b-474a51943d48","Type":"ContainerDied","Data":"f9ce5733f90ab37dd6011846ca9c8dd59626e8d4887ad81739d2db16f7140570"}
Oct 30 19:38:13.310775 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:13.310726    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-395ccc-zmhsj"
Oct 30 19:38:13.385291 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:13.384801    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be1be942-2abe-4c86-829b-474a51943d48-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be1be942-2abe-4c86-829b-474a51943d48-maas-api-openshift-ca\") pod \"be1be942-2abe-4c86-829b-474a51943d48\" (UID: \"be1be942-2abe-4c86-829b-474a51943d48\") "
Oct 30 19:38:13.385291 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:13.384906    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be1be942-2abe-4c86-829b-474a51943d48-kube-api-access-ltklq\" (UniqueName: \"kubernetes.io/projected/be1be942-2abe-4c86-829b-474a51943d48-kube-api-access-ltklq\") pod \"be1be942-2abe-4c86-829b-474a51943d48\" (UID: \"be1be942-2abe-4c86-829b-474a51943d48\") "
Oct 30 19:38:13.385291 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:13.384997    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be1be942-2abe-4c86-829b-474a51943d48-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be1be942-2abe-4c86-829b-474a51943d48-maas-api-xks-ca\") pod \"be1be942-2abe-4c86-829b-474a51943d48\" (UID: \"be1be942-2abe-4c86-829b-474a51943d48\") "
Oct 30 19:38:13.385291 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:13.385245    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be1be942-2abe-4c86-829b-474a51943d48-maas-api-openshift-ca" pod "be1be942-2abe-4c86-829b-474a51943d48" (UID: "be1be942-2abe-4c86-829b-474a51943d48"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:38:13.386436 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:13.386400    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be1be942-2abe-4c86-829b-474a51943d48-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:13.388278 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:13.388249    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be1be942-2abe-4c86-829b-474a51943d48-maas-api-xks-ca" pod "be1be942-2abe-4c86-829b-474a51943d48" (UID: "be1be942-2abe-4c86-829b-474a51943d48"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:38:13.388526 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:13.388506    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be1be942-2abe-4c86-829b-474a51943d48-kube-api-access-ltklq" pod "be1be942-2abe-4c86-829b-474a51943d48" (UID: "be1be942-2abe-4c86-829b-474a51943d48"). InnerVolumeSpecName "kube-api-access-ltklq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:13.488642 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:13.488599    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltklq\" (UniqueName: \"kubernetes.io/projected/be1be942-2abe-4c86-829b-474a51943d48-kube-api-access-ltklq\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:13.488642 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:13.488645    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be1be942-2abe-4c86-829b-474a51943d48-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:14.115714 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:14.115682    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-395ccc-zmhsj"
Oct 30 19:38:14.116020 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:14.115706    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-395ccc-zmhsj" event={"ID":"be1be942-2abe-4c86-829b-474a51943d48","Type":"ContainerDied","Data":"38d5be3b886b4b89ce841b7afa693c4839bfe834c10a86631d934be9227a11b4"}
Oct 30 19:38:14.116020 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:14.115975    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38d5be3b886b4b89ce841b7afa693c4839bfe834c10a86631d934be9227a11b4"
Oct 30 19:38:14.948664 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:14.948634    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-d1d256-5b65b66d9d-2djrn"
Oct 30 19:38:14.949145 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:14.948701    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d1d256-5b889bbcc5-qb7k8"
Oct 30 19:38:15.634565 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:15.634518    5043 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:38:15.634763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:15.634579    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr" podUID="6251d2e1-9d28-4bb8-a667-f415d0f79df0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:38:17.056516 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.056367    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf"
Oct 30 19:38:17.156727 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.156693    5043 generic.go:356] "Generic (PLEG): container finished" podID="6251d2e1-9d28-4bb8-a667-f415d0f79df0" containerID="c77d9dd0808935fc5a408dd9c5be241e32f2bdb681f4e19ccf52a5775b214346" exitCode=0
Oct 30 19:38:17.157042 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.157019    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr" event={"ID":"6251d2e1-9d28-4bb8-a667-f415d0f79df0","Type":"ContainerDied","Data":"c77d9dd0808935fc5a408dd9c5be241e32f2bdb681f4e19ccf52a5775b214346"}
Oct 30 19:38:17.157160 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.157080    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr" event={"ID":"6251d2e1-9d28-4bb8-a667-f415d0f79df0","Type":"ContainerDied","Data":"33778dc570b1aac3800b45d995940b6b45aa7cee3a513b0293061d8ace505b75"}
Oct 30 19:38:17.157160 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.157108    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33778dc570b1aac3800b45d995940b6b45aa7cee3a513b0293061d8ace505b75"
Oct 30 19:38:17.157160 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.157141    5043 scope.go:122] "RemoveContainer" containerID="d17141ca8ee5837753e4a10eddd4997368f9905465698c137e35f00095d4e7e3"
Oct 30 19:38:17.159038 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.159013    5043 generic.go:356] "Generic (PLEG): container finished" podID="a62f36bd-5442-4679-8616-5b75487cf90a" containerID="86e91c900aa0277e244ae2e463d59315e17fe21ca62e63c4573764dcbec73794" exitCode=0
Oct 30 19:38:17.159231 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.159074    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf"
Oct 30 19:38:17.159316 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.159091    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf" event={"ID":"a62f36bd-5442-4679-8616-5b75487cf90a","Type":"ContainerDied","Data":"86e91c900aa0277e244ae2e463d59315e17fe21ca62e63c4573764dcbec73794"}
Oct 30 19:38:17.159391 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.159369    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf" event={"ID":"a62f36bd-5442-4679-8616-5b75487cf90a","Type":"ContainerDied","Data":"7d7878d9134ee3ff18cabff8749a63ef222b76f3d10d6fce92bf97798baac2c0"}
Oct 30 19:38:17.161789 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.161231    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr"
Oct 30 19:38:17.172690 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.172662    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a62f36bd-5442-4679-8616-5b75487cf90a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a62f36bd-5442-4679-8616-5b75487cf90a-workload-socket\") pod \"a62f36bd-5442-4679-8616-5b75487cf90a\" (UID: \"a62f36bd-5442-4679-8616-5b75487cf90a\") "
Oct 30 19:38:17.172888 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.172772    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a62f36bd-5442-4679-8616-5b75487cf90a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a62f36bd-5442-4679-8616-5b75487cf90a-istio-podinfo\") pod \"a62f36bd-5442-4679-8616-5b75487cf90a\" (UID: \"a62f36bd-5442-4679-8616-5b75487cf90a\") "
Oct 30 19:38:17.172888 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.172844    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a62f36bd-5442-4679-8616-5b75487cf90a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a62f36bd-5442-4679-8616-5b75487cf90a-istio-data\") pod \"a62f36bd-5442-4679-8616-5b75487cf90a\" (UID: \"a62f36bd-5442-4679-8616-5b75487cf90a\") "
Oct 30 19:38:17.172888 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.172881    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a62f36bd-5442-4679-8616-5b75487cf90a-kube-api-access-rq6fh\" (UniqueName: \"kubernetes.io/projected/a62f36bd-5442-4679-8616-5b75487cf90a-kube-api-access-rq6fh\") pod \"a62f36bd-5442-4679-8616-5b75487cf90a\" (UID: \"a62f36bd-5442-4679-8616-5b75487cf90a\") "
Oct 30 19:38:17.173055 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.172922    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a62f36bd-5442-4679-8616-5b75487cf90a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a62f36bd-5442-4679-8616-5b75487cf90a-credential-socket\") pod \"a62f36bd-5442-4679-8616-5b75487cf90a\" (UID: \"a62f36bd-5442-4679-8616-5b75487cf90a\") "
Oct 30 19:38:17.173055 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.172958    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a62f36bd-5442-4679-8616-5b75487cf90a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a62f36bd-5442-4679-8616-5b75487cf90a-workload-certs\") pod \"a62f36bd-5442-4679-8616-5b75487cf90a\" (UID: \"a62f36bd-5442-4679-8616-5b75487cf90a\") "
Oct 30 19:38:17.173055 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.173017    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a62f36bd-5442-4679-8616-5b75487cf90a-istio-token\" (UniqueName: \"kubernetes.io/projected/a62f36bd-5442-4679-8616-5b75487cf90a-istio-token\") pod \"a62f36bd-5442-4679-8616-5b75487cf90a\" (UID: \"a62f36bd-5442-4679-8616-5b75487cf90a\") "
Oct 30 19:38:17.173206 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.173105    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a62f36bd-5442-4679-8616-5b75487cf90a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a62f36bd-5442-4679-8616-5b75487cf90a-istiod-ca-cert\") pod \"a62f36bd-5442-4679-8616-5b75487cf90a\" (UID: \"a62f36bd-5442-4679-8616-5b75487cf90a\") "
Oct 30 19:38:17.173206 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.173160    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a62f36bd-5442-4679-8616-5b75487cf90a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a62f36bd-5442-4679-8616-5b75487cf90a-istio-envoy\") pod \"a62f36bd-5442-4679-8616-5b75487cf90a\" (UID: \"a62f36bd-5442-4679-8616-5b75487cf90a\") "
Oct 30 19:38:17.174718 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.174562    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a62f36bd-5442-4679-8616-5b75487cf90a-workload-certs" pod "a62f36bd-5442-4679-8616-5b75487cf90a" (UID: "a62f36bd-5442-4679-8616-5b75487cf90a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:17.174870 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.174848    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a62f36bd-5442-4679-8616-5b75487cf90a-credential-socket" pod "a62f36bd-5442-4679-8616-5b75487cf90a" (UID: "a62f36bd-5442-4679-8616-5b75487cf90a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:17.175495 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.175366    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a62f36bd-5442-4679-8616-5b75487cf90a-istio-data" pod "a62f36bd-5442-4679-8616-5b75487cf90a" (UID: "a62f36bd-5442-4679-8616-5b75487cf90a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:17.175495 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.175446    5043 scope.go:122] "RemoveContainer" containerID="86e91c900aa0277e244ae2e463d59315e17fe21ca62e63c4573764dcbec73794"
Oct 30 19:38:17.175495 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.175445    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a62f36bd-5442-4679-8616-5b75487cf90a-istiod-ca-cert" pod "a62f36bd-5442-4679-8616-5b75487cf90a" (UID: "a62f36bd-5442-4679-8616-5b75487cf90a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:38:17.175813 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.175767    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a62f36bd-5442-4679-8616-5b75487cf90a-workload-socket" pod "a62f36bd-5442-4679-8616-5b75487cf90a" (UID: "a62f36bd-5442-4679-8616-5b75487cf90a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:17.179091 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.179062    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a62f36bd-5442-4679-8616-5b75487cf90a-istio-token" pod "a62f36bd-5442-4679-8616-5b75487cf90a" (UID: "a62f36bd-5442-4679-8616-5b75487cf90a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:17.182055 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.182028    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a62f36bd-5442-4679-8616-5b75487cf90a-istio-podinfo" pod "a62f36bd-5442-4679-8616-5b75487cf90a" (UID: "a62f36bd-5442-4679-8616-5b75487cf90a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:38:17.184666 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.184637    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a62f36bd-5442-4679-8616-5b75487cf90a-istio-envoy" pod "a62f36bd-5442-4679-8616-5b75487cf90a" (UID: "a62f36bd-5442-4679-8616-5b75487cf90a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:17.186835 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.186694    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a62f36bd-5442-4679-8616-5b75487cf90a-kube-api-access-rq6fh" pod "a62f36bd-5442-4679-8616-5b75487cf90a" (UID: "a62f36bd-5442-4679-8616-5b75487cf90a"). InnerVolumeSpecName "kube-api-access-rq6fh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:17.198556 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.198532    5043 scope.go:122] "RemoveContainer" containerID="86e91c900aa0277e244ae2e463d59315e17fe21ca62e63c4573764dcbec73794"
Oct 30 19:38:17.199064 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:38:17.198998    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86e91c900aa0277e244ae2e463d59315e17fe21ca62e63c4573764dcbec73794\": container with ID starting with 86e91c900aa0277e244ae2e463d59315e17fe21ca62e63c4573764dcbec73794 not found: ID does not exist" containerID="86e91c900aa0277e244ae2e463d59315e17fe21ca62e63c4573764dcbec73794"
Oct 30 19:38:17.199064 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.199035    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86e91c900aa0277e244ae2e463d59315e17fe21ca62e63c4573764dcbec73794"} err="failed to get container status \"86e91c900aa0277e244ae2e463d59315e17fe21ca62e63c4573764dcbec73794\": rpc error: code = NotFound desc = could not find container \"86e91c900aa0277e244ae2e463d59315e17fe21ca62e63c4573764dcbec73794\": container with ID starting with 86e91c900aa0277e244ae2e463d59315e17fe21ca62e63c4573764dcbec73794 not found: ID does not exist"
Oct 30 19:38:17.274576 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.274542    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6251d2e1-9d28-4bb8-a667-f415d0f79df0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6251d2e1-9d28-4bb8-a667-f415d0f79df0-credential-socket\") pod \"6251d2e1-9d28-4bb8-a667-f415d0f79df0\" (UID: \"6251d2e1-9d28-4bb8-a667-f415d0f79df0\") "
Oct 30 19:38:17.274576 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.274583    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6251d2e1-9d28-4bb8-a667-f415d0f79df0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6251d2e1-9d28-4bb8-a667-f415d0f79df0-istiod-ca-cert\") pod \"6251d2e1-9d28-4bb8-a667-f415d0f79df0\" (UID: \"6251d2e1-9d28-4bb8-a667-f415d0f79df0\") "
Oct 30 19:38:17.274842 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.274617    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6251d2e1-9d28-4bb8-a667-f415d0f79df0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6251d2e1-9d28-4bb8-a667-f415d0f79df0-istio-podinfo\") pod \"6251d2e1-9d28-4bb8-a667-f415d0f79df0\" (UID: \"6251d2e1-9d28-4bb8-a667-f415d0f79df0\") "
Oct 30 19:38:17.274842 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.274661    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6251d2e1-9d28-4bb8-a667-f415d0f79df0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6251d2e1-9d28-4bb8-a667-f415d0f79df0-workload-certs\") pod \"6251d2e1-9d28-4bb8-a667-f415d0f79df0\" (UID: \"6251d2e1-9d28-4bb8-a667-f415d0f79df0\") "
Oct 30 19:38:17.274842 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.274695    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6251d2e1-9d28-4bb8-a667-f415d0f79df0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6251d2e1-9d28-4bb8-a667-f415d0f79df0-istio-envoy\") pod \"6251d2e1-9d28-4bb8-a667-f415d0f79df0\" (UID: \"6251d2e1-9d28-4bb8-a667-f415d0f79df0\") "
Oct 30 19:38:17.274982 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.274886    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6251d2e1-9d28-4bb8-a667-f415d0f79df0-credential-socket" pod "6251d2e1-9d28-4bb8-a667-f415d0f79df0" (UID: "6251d2e1-9d28-4bb8-a667-f415d0f79df0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:17.274982 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.274895    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6251d2e1-9d28-4bb8-a667-f415d0f79df0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6251d2e1-9d28-4bb8-a667-f415d0f79df0-istio-data\") pod \"6251d2e1-9d28-4bb8-a667-f415d0f79df0\" (UID: \"6251d2e1-9d28-4bb8-a667-f415d0f79df0\") "
Oct 30 19:38:17.275075 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.274980    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6251d2e1-9d28-4bb8-a667-f415d0f79df0-kube-api-access-2w4kq\" (UniqueName: \"kubernetes.io/projected/6251d2e1-9d28-4bb8-a667-f415d0f79df0-kube-api-access-2w4kq\") pod \"6251d2e1-9d28-4bb8-a667-f415d0f79df0\" (UID: \"6251d2e1-9d28-4bb8-a667-f415d0f79df0\") "
Oct 30 19:38:17.275075 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.275014    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6251d2e1-9d28-4bb8-a667-f415d0f79df0-istio-token\" (UniqueName: \"kubernetes.io/projected/6251d2e1-9d28-4bb8-a667-f415d0f79df0-istio-token\") pod \"6251d2e1-9d28-4bb8-a667-f415d0f79df0\" (UID: \"6251d2e1-9d28-4bb8-a667-f415d0f79df0\") "
Oct 30 19:38:17.275075 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.275026    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6251d2e1-9d28-4bb8-a667-f415d0f79df0-istiod-ca-cert" pod "6251d2e1-9d28-4bb8-a667-f415d0f79df0" (UID: "6251d2e1-9d28-4bb8-a667-f415d0f79df0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:38:17.275209 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.275182    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6251d2e1-9d28-4bb8-a667-f415d0f79df0-workload-certs" pod "6251d2e1-9d28-4bb8-a667-f415d0f79df0" (UID: "6251d2e1-9d28-4bb8-a667-f415d0f79df0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:17.275209 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.275198    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6251d2e1-9d28-4bb8-a667-f415d0f79df0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6251d2e1-9d28-4bb8-a667-f415d0f79df0-workload-socket\") pod \"6251d2e1-9d28-4bb8-a667-f415d0f79df0\" (UID: \"6251d2e1-9d28-4bb8-a667-f415d0f79df0\") "
Oct 30 19:38:17.275786 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.275420    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6251d2e1-9d28-4bb8-a667-f415d0f79df0-workload-socket" pod "6251d2e1-9d28-4bb8-a667-f415d0f79df0" (UID: "6251d2e1-9d28-4bb8-a667-f415d0f79df0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:17.275786 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.275524    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6251d2e1-9d28-4bb8-a667-f415d0f79df0-istio-data" pod "6251d2e1-9d28-4bb8-a667-f415d0f79df0" (UID: "6251d2e1-9d28-4bb8-a667-f415d0f79df0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:17.276341 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.276320    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6251d2e1-9d28-4bb8-a667-f415d0f79df0-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:17.276401 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.276348    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a62f36bd-5442-4679-8616-5b75487cf90a-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:17.276401 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.276364    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a62f36bd-5442-4679-8616-5b75487cf90a-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:17.276401 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.276379    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a62f36bd-5442-4679-8616-5b75487cf90a-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:17.276401 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.276393    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq6fh\" (UniqueName: \"kubernetes.io/projected/a62f36bd-5442-4679-8616-5b75487cf90a-kube-api-access-rq6fh\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:17.276592 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.276409    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a62f36bd-5442-4679-8616-5b75487cf90a-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:17.276592 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.276424    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a62f36bd-5442-4679-8616-5b75487cf90a-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:17.276592 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.276438    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6251d2e1-9d28-4bb8-a667-f415d0f79df0-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:17.276592 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.276453    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a62f36bd-5442-4679-8616-5b75487cf90a-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:17.276592 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.276466    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a62f36bd-5442-4679-8616-5b75487cf90a-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:17.276592 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.276479    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a62f36bd-5442-4679-8616-5b75487cf90a-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:17.276592 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.276492    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6251d2e1-9d28-4bb8-a667-f415d0f79df0-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:17.276592 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.276505    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6251d2e1-9d28-4bb8-a667-f415d0f79df0-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:17.276592 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.276519    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6251d2e1-9d28-4bb8-a667-f415d0f79df0-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:17.279248 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.279162    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6251d2e1-9d28-4bb8-a667-f415d0f79df0-istio-podinfo" pod "6251d2e1-9d28-4bb8-a667-f415d0f79df0" (UID: "6251d2e1-9d28-4bb8-a667-f415d0f79df0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:38:17.279248 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.279169    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6251d2e1-9d28-4bb8-a667-f415d0f79df0-istio-envoy" pod "6251d2e1-9d28-4bb8-a667-f415d0f79df0" (UID: "6251d2e1-9d28-4bb8-a667-f415d0f79df0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:17.279248 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.279213    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6251d2e1-9d28-4bb8-a667-f415d0f79df0-kube-api-access-2w4kq" pod "6251d2e1-9d28-4bb8-a667-f415d0f79df0" (UID: "6251d2e1-9d28-4bb8-a667-f415d0f79df0"). InnerVolumeSpecName "kube-api-access-2w4kq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:17.279459 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.279391    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6251d2e1-9d28-4bb8-a667-f415d0f79df0-istio-token" pod "6251d2e1-9d28-4bb8-a667-f415d0f79df0" (UID: "6251d2e1-9d28-4bb8-a667-f415d0f79df0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:17.378440 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.378405    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6251d2e1-9d28-4bb8-a667-f415d0f79df0-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:17.378440 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.378437    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6251d2e1-9d28-4bb8-a667-f415d0f79df0-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:17.378626 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.378450    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w4kq\" (UniqueName: \"kubernetes.io/projected/6251d2e1-9d28-4bb8-a667-f415d0f79df0-kube-api-access-2w4kq\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:17.378626 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.378464    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6251d2e1-9d28-4bb8-a667-f415d0f79df0-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:17.568759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.564781    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf"]
Oct 30 19:38:17.592759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:17.583434    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-2dmhf"]
Oct 30 19:38:18.172863 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:18.172827    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr"
Oct 30 19:38:18.256164 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:18.256128    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr"]
Oct 30 19:38:18.275561 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:18.275505    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-jggxr"]
Oct 30 19:38:18.649686 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:18.649646    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6251d2e1-9d28-4bb8-a667-f415d0f79df0" path="/var/lib/kubelet/pods/6251d2e1-9d28-4bb8-a667-f415d0f79df0/volumes"
Oct 30 19:38:18.650416 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:18.650396    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a62f36bd-5442-4679-8616-5b75487cf90a" path="/var/lib/kubelet/pods/a62f36bd-5442-4679-8616-5b75487cf90a/volumes"
Oct 30 19:38:19.786653 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:19.786620    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-395ccc-7d4b79d8f9-gn47s"]
Oct 30 19:38:23.720161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:23.720132    5043 scope.go:122] "RemoveContainer" containerID="3af35e38edcbc548d2ab34f0c064ae1232d89a9d115d6e10fd0e3ec1cfa1611c"
Oct 30 19:38:23.731464 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:23.731442    5043 scope.go:122] "RemoveContainer" containerID="0776b546f768c72deec0cf46a07cc28d49ee2bc972d8fde2f7c62913aed24d25"
Oct 30 19:38:23.741663 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:23.741642    5043 scope.go:122] "RemoveContainer" containerID="c77d9dd0808935fc5a408dd9c5be241e32f2bdb681f4e19ccf52a5775b214346"
Oct 30 19:38:23.752026 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:23.752006    5043 scope.go:122] "RemoveContainer" containerID="e3a2ede7c20726db7d06d5bbfa60b1d825d1ff886ed5a5817b64fbeeac23bc75"
Oct 30 19:38:23.764174 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:23.764143    5043 scope.go:122] "RemoveContainer" containerID="e5a67b1d6e4ebd84400f9709701fa2befa9c18f1a355626c667f8d29c74a5698"
Oct 30 19:38:24.789622 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:24.787872    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-395ccc-7d4b79d8f9-gn47s" podUID="5b3cc611-c18a-49d7-a09c-5185d7ee2ea4" containerName="payload-pre-processing" containerID="cri-o://7d086db4b0c80017a7f15a80e4f4916a65195bd169eba686dd5ddb2716f5a698" gracePeriod=25
Oct 30 19:38:24.987179 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:38:24.987095    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:38:25.150007 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:25.149982    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-395ccc-7d4b79d8f9-gn47s"
Oct 30 19:38:25.226289 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:25.226252    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b3cc611-c18a-49d7-a09c-5185d7ee2ea4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b3cc611-c18a-49d7-a09c-5185d7ee2ea4-plugins-config-volume\") pod \"5b3cc611-c18a-49d7-a09c-5185d7ee2ea4\" (UID: \"5b3cc611-c18a-49d7-a09c-5185d7ee2ea4\") "
Oct 30 19:38:25.226503 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:25.226380    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b3cc611-c18a-49d7-a09c-5185d7ee2ea4-kube-api-access-2vqhw\" (UniqueName: \"kubernetes.io/projected/5b3cc611-c18a-49d7-a09c-5185d7ee2ea4-kube-api-access-2vqhw\") pod \"5b3cc611-c18a-49d7-a09c-5185d7ee2ea4\" (UID: \"5b3cc611-c18a-49d7-a09c-5185d7ee2ea4\") "
Oct 30 19:38:25.226849 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:25.226813    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b3cc611-c18a-49d7-a09c-5185d7ee2ea4-plugins-config-volume" pod "5b3cc611-c18a-49d7-a09c-5185d7ee2ea4" (UID: "5b3cc611-c18a-49d7-a09c-5185d7ee2ea4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:38:25.227416 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:25.227376    5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b3cc611-c18a-49d7-a09c-5185d7ee2ea4-plugins-config-volume\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:25.229115 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:25.229082    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b3cc611-c18a-49d7-a09c-5185d7ee2ea4-kube-api-access-2vqhw" pod "5b3cc611-c18a-49d7-a09c-5185d7ee2ea4" (UID: "5b3cc611-c18a-49d7-a09c-5185d7ee2ea4"). InnerVolumeSpecName "kube-api-access-2vqhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:25.306853 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:25.306821    5043 generic.go:356] "Generic (PLEG): container finished" podID="5b3cc611-c18a-49d7-a09c-5185d7ee2ea4" containerID="7d086db4b0c80017a7f15a80e4f4916a65195bd169eba686dd5ddb2716f5a698" exitCode=0
Oct 30 19:38:25.307048 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:25.306883    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-395ccc-7d4b79d8f9-gn47s"
Oct 30 19:38:25.307048 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:25.306925    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-395ccc-7d4b79d8f9-gn47s" event={"ID":"5b3cc611-c18a-49d7-a09c-5185d7ee2ea4","Type":"ContainerDied","Data":"7d086db4b0c80017a7f15a80e4f4916a65195bd169eba686dd5ddb2716f5a698"}
Oct 30 19:38:25.307048 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:25.306985    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-395ccc-7d4b79d8f9-gn47s" event={"ID":"5b3cc611-c18a-49d7-a09c-5185d7ee2ea4","Type":"ContainerDied","Data":"4e6da541b4566f74fd18a6c47600228ceda9a29d296d8580fee52b9891b9bf36"}
Oct 30 19:38:25.307048 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:25.307016    5043 scope.go:122] "RemoveContainer" containerID="7d086db4b0c80017a7f15a80e4f4916a65195bd169eba686dd5ddb2716f5a698"
Oct 30 19:38:25.319872 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:25.319851    5043 scope.go:122] "RemoveContainer" containerID="7d086db4b0c80017a7f15a80e4f4916a65195bd169eba686dd5ddb2716f5a698"
Oct 30 19:38:25.320171 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:38:25.320149    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d086db4b0c80017a7f15a80e4f4916a65195bd169eba686dd5ddb2716f5a698\": container with ID starting with 7d086db4b0c80017a7f15a80e4f4916a65195bd169eba686dd5ddb2716f5a698 not found: ID does not exist" containerID="7d086db4b0c80017a7f15a80e4f4916a65195bd169eba686dd5ddb2716f5a698"
Oct 30 19:38:25.320247 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:25.320179    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d086db4b0c80017a7f15a80e4f4916a65195bd169eba686dd5ddb2716f5a698"} err="failed to get container status \"7d086db4b0c80017a7f15a80e4f4916a65195bd169eba686dd5ddb2716f5a698\": rpc error: code = NotFound desc = could not find container \"7d086db4b0c80017a7f15a80e4f4916a65195bd169eba686dd5ddb2716f5a698\": container with ID starting with 7d086db4b0c80017a7f15a80e4f4916a65195bd169eba686dd5ddb2716f5a698 not found: ID does not exist"
Oct 30 19:38:25.328637 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:25.328609    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vqhw\" (UniqueName: \"kubernetes.io/projected/5b3cc611-c18a-49d7-a09c-5185d7ee2ea4-kube-api-access-2vqhw\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:25.333006 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:25.332982    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-395ccc-7d4b79d8f9-gn47s"]
Oct 30 19:38:25.342729 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:25.342665    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-395ccc-7d4b79d8f9-gn47s"]
Oct 30 19:38:26.647954 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:26.647923    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b3cc611-c18a-49d7-a09c-5185d7ee2ea4" path="/var/lib/kubelet/pods/5b3cc611-c18a-49d7-a09c-5185d7ee2ea4/volumes"
Oct 30 19:38:26.824060 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:26.824016    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx"]
Oct 30 19:38:26.825146 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:26.825125    5043 cpu_manager.go:414] "containerMap: removing container" podUID="a62f36bd-5442-4679-8616-5b75487cf90a" containerName="istio-proxy"
Oct 30 19:38:26.825146 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:26.825144    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a62f36bd-5442-4679-8616-5b75487cf90a" containerName="istio-proxy"
Oct 30 19:38:26.825350 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:26.825194    5043 cpu_manager.go:414] "containerMap: removing container" podUID="be1be942-2abe-4c86-829b-474a51943d48" containerName="revoke-keys"
Oct 30 19:38:26.825350 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:26.825204    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be1be942-2abe-4c86-829b-474a51943d48" containerName="revoke-keys"
Oct 30 19:38:26.825350 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:26.825222    5043 cpu_manager.go:414] "containerMap: removing container" podUID="6251d2e1-9d28-4bb8-a667-f415d0f79df0" containerName="istio-proxy"
Oct 30 19:38:26.825350 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:26.825229    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6251d2e1-9d28-4bb8-a667-f415d0f79df0" containerName="istio-proxy"
Oct 30 19:38:26.825350 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:26.825259    5043 cpu_manager.go:414] "containerMap: removing container" podUID="6251d2e1-9d28-4bb8-a667-f415d0f79df0" containerName="istio-proxy"
Oct 30 19:38:26.825350 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:26.825268    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6251d2e1-9d28-4bb8-a667-f415d0f79df0" containerName="istio-proxy"
Oct 30 19:38:26.825350 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:26.825280    5043 cpu_manager.go:414] "containerMap: removing container" podUID="5b3cc611-c18a-49d7-a09c-5185d7ee2ea4" containerName="payload-pre-processing"
Oct 30 19:38:26.825350 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:26.825287    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b3cc611-c18a-49d7-a09c-5185d7ee2ea4" containerName="payload-pre-processing"
Oct 30 19:38:26.825840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:26.825473    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="be1be942-2abe-4c86-829b-474a51943d48" containerName="revoke-keys"
Oct 30 19:38:26.825840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:26.825489    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="6251d2e1-9d28-4bb8-a667-f415d0f79df0" containerName="istio-proxy"
Oct 30 19:38:26.825840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:26.825501    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="6251d2e1-9d28-4bb8-a667-f415d0f79df0" containerName="istio-proxy"
Oct 30 19:38:26.825840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:26.825510    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b3cc611-c18a-49d7-a09c-5185d7ee2ea4" containerName="payload-pre-processing"
Oct 30 19:38:26.825840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:26.825526    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="a62f36bd-5442-4679-8616-5b75487cf90a" containerName="istio-proxy"
Oct 30 19:38:26.831259 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:26.831235    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx"
Oct 30 19:38:26.841912 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:26.841890    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-1e13df-openshift-default-dockercfg-tkpw2\""
Oct 30 19:38:26.845690 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:26.845670    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx"]
Oct 30 19:38:26.982252 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:26.982215    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f791cec1-a1e4-4067-9221-cc04dafba0eb-istio-token\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx\" (UID: \"f791cec1-a1e4-4067-9221-cc04dafba0eb\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx"
Oct 30 19:38:26.982441 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:26.982385    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f791cec1-a1e4-4067-9221-cc04dafba0eb-istiod-ca-cert\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx\" (UID: \"f791cec1-a1e4-4067-9221-cc04dafba0eb\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx"
Oct 30 19:38:26.982486 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:26.982467    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f791cec1-a1e4-4067-9221-cc04dafba0eb-workload-certs\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx\" (UID: \"f791cec1-a1e4-4067-9221-cc04dafba0eb\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx"
Oct 30 19:38:26.982540 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:26.982526    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f791cec1-a1e4-4067-9221-cc04dafba0eb-workload-socket\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx\" (UID: \"f791cec1-a1e4-4067-9221-cc04dafba0eb\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx"
Oct 30 19:38:26.982626 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:26.982597    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-899wx\" (UniqueName: \"kubernetes.io/projected/f791cec1-a1e4-4067-9221-cc04dafba0eb-kube-api-access-899wx\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx\" (UID: \"f791cec1-a1e4-4067-9221-cc04dafba0eb\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx"
Oct 30 19:38:26.982781 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:26.982728    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f791cec1-a1e4-4067-9221-cc04dafba0eb-credential-socket\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx\" (UID: \"f791cec1-a1e4-4067-9221-cc04dafba0eb\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx"
Oct 30 19:38:26.982843 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:26.982824    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f791cec1-a1e4-4067-9221-cc04dafba0eb-istio-data\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx\" (UID: \"f791cec1-a1e4-4067-9221-cc04dafba0eb\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx"
Oct 30 19:38:26.982913 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:26.982899    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f791cec1-a1e4-4067-9221-cc04dafba0eb-istio-envoy\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx\" (UID: \"f791cec1-a1e4-4067-9221-cc04dafba0eb\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx"
Oct 30 19:38:26.982953 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:26.982938    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f791cec1-a1e4-4067-9221-cc04dafba0eb-istio-podinfo\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx\" (UID: \"f791cec1-a1e4-4067-9221-cc04dafba0eb\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx"
Oct 30 19:38:27.109804 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:27.101981    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f791cec1-a1e4-4067-9221-cc04dafba0eb-istio-token\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx\" (UID: \"f791cec1-a1e4-4067-9221-cc04dafba0eb\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx"
Oct 30 19:38:27.109804 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:27.102136    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f791cec1-a1e4-4067-9221-cc04dafba0eb-istiod-ca-cert\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx\" (UID: \"f791cec1-a1e4-4067-9221-cc04dafba0eb\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx"
Oct 30 19:38:27.109804 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:27.102179    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f791cec1-a1e4-4067-9221-cc04dafba0eb-workload-certs\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx\" (UID: \"f791cec1-a1e4-4067-9221-cc04dafba0eb\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx"
Oct 30 19:38:27.109804 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:27.102236    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f791cec1-a1e4-4067-9221-cc04dafba0eb-workload-socket\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx\" (UID: \"f791cec1-a1e4-4067-9221-cc04dafba0eb\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx"
Oct 30 19:38:27.109804 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:27.102260    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-899wx\" (UniqueName: \"kubernetes.io/projected/f791cec1-a1e4-4067-9221-cc04dafba0eb-kube-api-access-899wx\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx\" (UID: \"f791cec1-a1e4-4067-9221-cc04dafba0eb\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx"
Oct 30 19:38:27.109804 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:27.102318    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f791cec1-a1e4-4067-9221-cc04dafba0eb-credential-socket\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx\" (UID: \"f791cec1-a1e4-4067-9221-cc04dafba0eb\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx"
Oct 30 19:38:27.109804 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:27.102359    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f791cec1-a1e4-4067-9221-cc04dafba0eb-istio-data\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx\" (UID: \"f791cec1-a1e4-4067-9221-cc04dafba0eb\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx"
Oct 30 19:38:27.109804 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:27.102478    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f791cec1-a1e4-4067-9221-cc04dafba0eb-istio-envoy\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx\" (UID: \"f791cec1-a1e4-4067-9221-cc04dafba0eb\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx"
Oct 30 19:38:27.109804 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:27.102534    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f791cec1-a1e4-4067-9221-cc04dafba0eb-istio-podinfo\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx\" (UID: \"f791cec1-a1e4-4067-9221-cc04dafba0eb\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx"
Oct 30 19:38:27.133760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:27.127477    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f791cec1-a1e4-4067-9221-cc04dafba0eb-workload-socket\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx\" (UID: \"f791cec1-a1e4-4067-9221-cc04dafba0eb\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx"
Oct 30 19:38:27.133760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:27.128443    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f791cec1-a1e4-4067-9221-cc04dafba0eb-istiod-ca-cert\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx\" (UID: \"f791cec1-a1e4-4067-9221-cc04dafba0eb\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx"
Oct 30 19:38:27.133760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:27.128677    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f791cec1-a1e4-4067-9221-cc04dafba0eb-workload-certs\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx\" (UID: \"f791cec1-a1e4-4067-9221-cc04dafba0eb\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx"
Oct 30 19:38:27.133760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:27.128925    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f791cec1-a1e4-4067-9221-cc04dafba0eb-credential-socket\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx\" (UID: \"f791cec1-a1e4-4067-9221-cc04dafba0eb\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx"
Oct 30 19:38:27.133760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:27.129374    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f791cec1-a1e4-4067-9221-cc04dafba0eb-istio-data\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx\" (UID: \"f791cec1-a1e4-4067-9221-cc04dafba0eb\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx"
Oct 30 19:38:27.156768 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:27.152844    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f791cec1-a1e4-4067-9221-cc04dafba0eb-istio-podinfo\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx\" (UID: \"f791cec1-a1e4-4067-9221-cc04dafba0eb\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx"
Oct 30 19:38:27.156768 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:27.155330    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f791cec1-a1e4-4067-9221-cc04dafba0eb-istio-envoy\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx\" (UID: \"f791cec1-a1e4-4067-9221-cc04dafba0eb\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx"
Oct 30 19:38:27.168772 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:27.168068    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-899wx\" (UniqueName: \"kubernetes.io/projected/f791cec1-a1e4-4067-9221-cc04dafba0eb-kube-api-access-899wx\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx\" (UID: \"f791cec1-a1e4-4067-9221-cc04dafba0eb\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx"
Oct 30 19:38:27.254801 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:27.254701    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f791cec1-a1e4-4067-9221-cc04dafba0eb-istio-token\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx\" (UID: \"f791cec1-a1e4-4067-9221-cc04dafba0eb\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx"
Oct 30 19:38:27.470047 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:27.470017    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx"
Oct 30 19:38:27.780534 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:27.780494    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx"]
Oct 30 19:38:27.813766 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:38:27.811450    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf791cec1_a1e4_4067_9221_cc04dafba0eb.slice/crio-f9741431c94b72b493449574cba873714903d28bd849f67b1fcc40a0022c2ffa WatchSource:0}: Error finding container f9741431c94b72b493449574cba873714903d28bd849f67b1fcc40a0022c2ffa: Status 404 returned error can't find the container with id f9741431c94b72b493449574cba873714903d28bd849f67b1fcc40a0022c2ffa
Oct 30 19:38:27.857447 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:27.857392    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:38:27.857764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:27.857729    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:38:27.857932 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:27.857915    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:38:28.108519 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:28.108487    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-395ccc-zmhsj"]
Oct 30 19:38:28.113453 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:28.113421    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-395ccc-zmhsj"]
Oct 30 19:38:28.345639 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:28.345554    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx" event={"ID":"f791cec1-a1e4-4067-9221-cc04dafba0eb","Type":"ContainerStarted","Data":"2bf0d118cce4ea36dc19bc4338eaf9c3cf4336991c115f7aa305eedaa4b76956"}
Oct 30 19:38:28.345639 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:28.345607    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx" event={"ID":"f791cec1-a1e4-4067-9221-cc04dafba0eb","Type":"ContainerStarted","Data":"f9741431c94b72b493449574cba873714903d28bd849f67b1fcc40a0022c2ffa"}
Oct 30 19:38:28.412674 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:28.412607    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx" podStartSLOduration=2.412584642 podStartE2EDuration="2.412584642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:28.382090942 +0000 UTC m=+1327.893150908" watchObservedRunningTime="2026-10-30 19:38:28.412584642 +0000 UTC m=+1327.923644608"
Oct 30 19:38:28.470666 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:28.470638    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx"
Oct 30 19:38:28.648690 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:28.648603    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be1be942-2abe-4c86-829b-474a51943d48" path="/var/lib/kubelet/pods/be1be942-2abe-4c86-829b-474a51943d48/volumes"
Oct 30 19:38:29.476934 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:29.476902    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx"
Oct 30 19:38:30.365706 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:30.365679    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx"
Oct 30 19:38:30.366426 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:30.366409    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx"
Oct 30 19:38:32.055587 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:32.055535    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1e13df-78b7ccf4bc-8ldn2"]
Oct 30 19:38:32.062299 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:32.062269    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1e13df-78b7ccf4bc-8ldn2"
Oct 30 19:38:32.074434 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:32.073345    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-1e13df\""
Oct 30 19:38:32.074434 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:32.073361    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-1e13df-dockercfg-bgd85\""
Oct 30 19:38:32.078714 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:32.078480    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1e13df-78b7ccf4bc-8ldn2"]
Oct 30 19:38:32.223771 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:32.221887    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhrbb\" (UniqueName: \"kubernetes.io/projected/3cd1e9e6-6a23-40d8-aa51-0266d9c61446-kube-api-access-bhrbb\") pod \"payload-pre-processing-e2e-inf-b-1e13df-78b7ccf4bc-8ldn2\" (UID: \"3cd1e9e6-6a23-40d8-aa51-0266d9c61446\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1e13df-78b7ccf4bc-8ldn2"
Oct 30 19:38:32.223771 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:32.222136    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cd1e9e6-6a23-40d8-aa51-0266d9c61446-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-1e13df-78b7ccf4bc-8ldn2\" (UID: \"3cd1e9e6-6a23-40d8-aa51-0266d9c61446\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1e13df-78b7ccf4bc-8ldn2"
Oct 30 19:38:32.335808 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:32.325900    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhrbb\" (UniqueName: \"kubernetes.io/projected/3cd1e9e6-6a23-40d8-aa51-0266d9c61446-kube-api-access-bhrbb\") pod \"payload-pre-processing-e2e-inf-b-1e13df-78b7ccf4bc-8ldn2\" (UID: \"3cd1e9e6-6a23-40d8-aa51-0266d9c61446\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1e13df-78b7ccf4bc-8ldn2"
Oct 30 19:38:32.335808 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:32.326617    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cd1e9e6-6a23-40d8-aa51-0266d9c61446-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-1e13df-78b7ccf4bc-8ldn2\" (UID: \"3cd1e9e6-6a23-40d8-aa51-0266d9c61446\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1e13df-78b7ccf4bc-8ldn2"
Oct 30 19:38:32.335808 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:32.327561    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cd1e9e6-6a23-40d8-aa51-0266d9c61446-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-1e13df-78b7ccf4bc-8ldn2\" (UID: \"3cd1e9e6-6a23-40d8-aa51-0266d9c61446\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1e13df-78b7ccf4bc-8ldn2"
Oct 30 19:38:32.363755 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:32.356295    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhrbb\" (UniqueName: \"kubernetes.io/projected/3cd1e9e6-6a23-40d8-aa51-0266d9c61446-kube-api-access-bhrbb\") pod \"payload-pre-processing-e2e-inf-b-1e13df-78b7ccf4bc-8ldn2\" (UID: \"3cd1e9e6-6a23-40d8-aa51-0266d9c61446\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1e13df-78b7ccf4bc-8ldn2"
Oct 30 19:38:32.391002 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:32.385012    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1e13df-78b7ccf4bc-8ldn2"
Oct 30 19:38:32.838706 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:32.837857    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1e13df-78b7ccf4bc-8ldn2"]
Oct 30 19:38:33.516229 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:33.516190    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1e13df-78b7ccf4bc-8ldn2" event={"ID":"3cd1e9e6-6a23-40d8-aa51-0266d9c61446","Type":"ContainerStarted","Data":"a214ad1cb247e0cbbae5d35dd948109e53bdc09f70b4d3c45c691db8f7ec9d41"}
Oct 30 19:38:33.516776 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:33.516264    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1e13df-78b7ccf4bc-8ldn2"
Oct 30 19:38:33.516776 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:33.516288    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1e13df-78b7ccf4bc-8ldn2" event={"ID":"3cd1e9e6-6a23-40d8-aa51-0266d9c61446","Type":"ContainerStarted","Data":"d6f00b150c12a7505e8d9ef27ad9441cbe92df189fd2494e7e0074b3971332a4"}
Oct 30 19:38:33.538698 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:33.538643    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1e13df-78b7ccf4bc-8ldn2" podStartSLOduration=1.538628048 podStartE2EDuration="1.538628048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:33.534131825 +0000 UTC m=+1333.045191792" watchObservedRunningTime="2026-10-30 19:38:33.538628048 +0000 UTC m=+1333.049688012"
Oct 30 19:38:37.059243 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:37.059212    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6"]
Oct 30 19:38:37.059774 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:37.059433    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6" podUID="0615f18f-9ed4-4d64-bc4f-1a57e15286cf" containerName="istio-proxy" containerID="cri-o://f5dd80eab41e12d10a569be0d78c28bb4ebd90b8ce71880650c50736fc81e279" gracePeriod=30
Oct 30 19:38:37.576550 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:37.576519    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ae425f-dzwmr"]
Oct 30 19:38:37.581218 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:37.581197    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ae425f-dzwmr"
Oct 30 19:38:37.588023 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:37.587996    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:38:37.589252 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:37.588300    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:38:37.591040 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:37.591016    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ae425f-dzwmr"]
Oct 30 19:38:37.642854 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:37.642821    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:38:37.643106 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:37.643090    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:38:37.693610 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:37.693573    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2tj5\" (UniqueName: \"kubernetes.io/projected/346d6503-7d76-47c4-8528-18d1f0669680-kube-api-access-r2tj5\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-ae425f-dzwmr\" (UID: \"346d6503-7d76-47c4-8528-18d1f0669680\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ae425f-dzwmr"
Oct 30 19:38:37.693820 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:37.693639    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/346d6503-7d76-47c4-8528-18d1f0669680-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-ae425f-dzwmr\" (UID: \"346d6503-7d76-47c4-8528-18d1f0669680\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ae425f-dzwmr"
Oct 30 19:38:37.693973 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:37.693946    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/346d6503-7d76-47c4-8528-18d1f0669680-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-ae425f-dzwmr\" (UID: \"346d6503-7d76-47c4-8528-18d1f0669680\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ae425f-dzwmr"
Oct 30 19:38:37.800215 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:37.800171    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2tj5\" (UniqueName: \"kubernetes.io/projected/346d6503-7d76-47c4-8528-18d1f0669680-kube-api-access-r2tj5\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-ae425f-dzwmr\" (UID: \"346d6503-7d76-47c4-8528-18d1f0669680\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ae425f-dzwmr"
Oct 30 19:38:37.801372 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:37.801344    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/346d6503-7d76-47c4-8528-18d1f0669680-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-ae425f-dzwmr\" (UID: \"346d6503-7d76-47c4-8528-18d1f0669680\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ae425f-dzwmr"
Oct 30 19:38:37.801911 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:37.801880    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/346d6503-7d76-47c4-8528-18d1f0669680-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-ae425f-dzwmr\" (UID: \"346d6503-7d76-47c4-8528-18d1f0669680\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ae425f-dzwmr"
Oct 30 19:38:37.822444 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:37.802617    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/346d6503-7d76-47c4-8528-18d1f0669680-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-ae425f-dzwmr\" (UID: \"346d6503-7d76-47c4-8528-18d1f0669680\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ae425f-dzwmr"
Oct 30 19:38:37.868830 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:37.830062    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/346d6503-7d76-47c4-8528-18d1f0669680-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-ae425f-dzwmr\" (UID: \"346d6503-7d76-47c4-8528-18d1f0669680\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ae425f-dzwmr"
Oct 30 19:38:37.868830 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:37.865409    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2tj5\" (UniqueName: \"kubernetes.io/projected/346d6503-7d76-47c4-8528-18d1f0669680-kube-api-access-r2tj5\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-ae425f-dzwmr\" (UID: \"346d6503-7d76-47c4-8528-18d1f0669680\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ae425f-dzwmr"
Oct 30 19:38:37.903475 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:37.898983    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ae425f-dzwmr"
Oct 30 19:38:38.283759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:38.276123    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ae425f-dzwmr"]
Oct 30 19:38:38.565587 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:38.565498    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ae425f-dzwmr" event={"ID":"346d6503-7d76-47c4-8528-18d1f0669680","Type":"ContainerStarted","Data":"3e7965175c30b9cfb326df2e71d3eecdf9388d31ebc664ca5cce6855e3b6fdac"}
Oct 30 19:38:39.577968 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:39.577921    5043 generic.go:356] "Generic (PLEG): container finished" podID="346d6503-7d76-47c4-8528-18d1f0669680" containerID="248e4b90b3b646f84dc3786f29009e13debc8f43c524da7dbad05def5d30719e" exitCode=0
Oct 30 19:38:39.578623 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:39.578025    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ae425f-dzwmr" event={"ID":"346d6503-7d76-47c4-8528-18d1f0669680","Type":"ContainerDied","Data":"248e4b90b3b646f84dc3786f29009e13debc8f43c524da7dbad05def5d30719e"}
Oct 30 19:38:40.753516 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:40.753483    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82012b-xc65q"]
Oct 30 19:38:40.760217 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:40.760187    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82012b-xc65q"
Oct 30 19:38:40.801230 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:40.798248    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82012b-xc65q"]
Oct 30 19:38:40.815170 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:40.804682    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c267fa9-6d83-4c9e-a9fb-10885801a21f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-82012b-xc65q\" (UID: \"5c267fa9-6d83-4c9e-a9fb-10885801a21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82012b-xc65q"
Oct 30 19:38:40.815170 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:40.814226    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c267fa9-6d83-4c9e-a9fb-10885801a21f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-82012b-xc65q\" (UID: \"5c267fa9-6d83-4c9e-a9fb-10885801a21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82012b-xc65q"
Oct 30 19:38:40.815170 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:40.814545    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5f8h\" (UniqueName: \"kubernetes.io/projected/5c267fa9-6d83-4c9e-a9fb-10885801a21f-kube-api-access-c5f8h\") pod \"maas-api-revoke-keys-e2e-api-a-82012b-xc65q\" (UID: \"5c267fa9-6d83-4c9e-a9fb-10885801a21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82012b-xc65q"
Oct 30 19:38:40.911418 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:40.911352    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ae425f-dzwmr"
Oct 30 19:38:40.915999 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:40.915968    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c267fa9-6d83-4c9e-a9fb-10885801a21f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-82012b-xc65q\" (UID: \"5c267fa9-6d83-4c9e-a9fb-10885801a21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82012b-xc65q"
Oct 30 19:38:40.916184 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:40.916166    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c267fa9-6d83-4c9e-a9fb-10885801a21f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-82012b-xc65q\" (UID: \"5c267fa9-6d83-4c9e-a9fb-10885801a21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82012b-xc65q"
Oct 30 19:38:40.916322 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:40.916303    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5f8h\" (UniqueName: \"kubernetes.io/projected/5c267fa9-6d83-4c9e-a9fb-10885801a21f-kube-api-access-c5f8h\") pod \"maas-api-revoke-keys-e2e-api-a-82012b-xc65q\" (UID: \"5c267fa9-6d83-4c9e-a9fb-10885801a21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82012b-xc65q"
Oct 30 19:38:40.916556 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:40.916515    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c267fa9-6d83-4c9e-a9fb-10885801a21f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-82012b-xc65q\" (UID: \"5c267fa9-6d83-4c9e-a9fb-10885801a21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82012b-xc65q"
Oct 30 19:38:40.918670 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:40.918646    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c267fa9-6d83-4c9e-a9fb-10885801a21f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-82012b-xc65q\" (UID: \"5c267fa9-6d83-4c9e-a9fb-10885801a21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82012b-xc65q"
Oct 30 19:38:40.927048 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:40.927030    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5f8h\" (UniqueName: \"kubernetes.io/projected/5c267fa9-6d83-4c9e-a9fb-10885801a21f-kube-api-access-c5f8h\") pod \"maas-api-revoke-keys-e2e-api-a-82012b-xc65q\" (UID: \"5c267fa9-6d83-4c9e-a9fb-10885801a21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82012b-xc65q"
Oct 30 19:38:41.032387 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:41.030413    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/346d6503-7d76-47c4-8528-18d1f0669680-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/346d6503-7d76-47c4-8528-18d1f0669680-maas-api-xks-ca\") pod \"346d6503-7d76-47c4-8528-18d1f0669680\" (UID: \"346d6503-7d76-47c4-8528-18d1f0669680\") "
Oct 30 19:38:41.032387 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:41.030582    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/346d6503-7d76-47c4-8528-18d1f0669680-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/346d6503-7d76-47c4-8528-18d1f0669680-maas-api-openshift-ca\") pod \"346d6503-7d76-47c4-8528-18d1f0669680\" (UID: \"346d6503-7d76-47c4-8528-18d1f0669680\") "
Oct 30 19:38:41.032387 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:41.030634    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/346d6503-7d76-47c4-8528-18d1f0669680-kube-api-access-r2tj5\" (UniqueName: \"kubernetes.io/projected/346d6503-7d76-47c4-8528-18d1f0669680-kube-api-access-r2tj5\") pod \"346d6503-7d76-47c4-8528-18d1f0669680\" (UID: \"346d6503-7d76-47c4-8528-18d1f0669680\") "
Oct 30 19:38:41.032387 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:41.032453    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/346d6503-7d76-47c4-8528-18d1f0669680-maas-api-openshift-ca" pod "346d6503-7d76-47c4-8528-18d1f0669680" (UID: "346d6503-7d76-47c4-8528-18d1f0669680"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:38:41.032387 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:41.037325    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/346d6503-7d76-47c4-8528-18d1f0669680-maas-api-xks-ca" pod "346d6503-7d76-47c4-8528-18d1f0669680" (UID: "346d6503-7d76-47c4-8528-18d1f0669680"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:38:41.039953 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:41.038461    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/346d6503-7d76-47c4-8528-18d1f0669680-kube-api-access-r2tj5" pod "346d6503-7d76-47c4-8528-18d1f0669680" (UID: "346d6503-7d76-47c4-8528-18d1f0669680"). InnerVolumeSpecName "kube-api-access-r2tj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:41.120526 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:41.120478    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82012b-xc65q"
Oct 30 19:38:41.133333 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:41.133303    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2tj5\" (UniqueName: \"kubernetes.io/projected/346d6503-7d76-47c4-8528-18d1f0669680-kube-api-access-r2tj5\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:41.133333 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:41.133331    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/346d6503-7d76-47c4-8528-18d1f0669680-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:41.133523 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:41.133341    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/346d6503-7d76-47c4-8528-18d1f0669680-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:41.418966 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:38:41.418935    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c267fa9_6d83_4c9e_a9fb_10885801a21f.slice/crio-8bf4aec55bd1f5459eaf782f5f7113dd20192608d1c091f642f376c378a6a9c8 WatchSource:0}: Error finding container 8bf4aec55bd1f5459eaf782f5f7113dd20192608d1c091f642f376c378a6a9c8: Status 404 returned error can't find the container with id 8bf4aec55bd1f5459eaf782f5f7113dd20192608d1c091f642f376c378a6a9c8
Oct 30 19:38:41.420533 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:41.420505    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82012b-xc65q"]
Oct 30 19:38:41.639673 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:41.639604    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ae425f-dzwmr" event={"ID":"346d6503-7d76-47c4-8528-18d1f0669680","Type":"ContainerDied","Data":"3e7965175c30b9cfb326df2e71d3eecdf9388d31ebc664ca5cce6855e3b6fdac"}
Oct 30 19:38:41.639673 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:41.639673    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e7965175c30b9cfb326df2e71d3eecdf9388d31ebc664ca5cce6855e3b6fdac"
Oct 30 19:38:41.641279 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:41.641256    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82012b-xc65q" event={"ID":"5c267fa9-6d83-4c9e-a9fb-10885801a21f","Type":"ContainerStarted","Data":"8bf4aec55bd1f5459eaf782f5f7113dd20192608d1c091f642f376c378a6a9c8"}
Oct 30 19:38:41.648924 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:41.648901    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ae425f-dzwmr"
Oct 30 19:38:42.653683 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:42.653644    5043 generic.go:356] "Generic (PLEG): container finished" podID="5c267fa9-6d83-4c9e-a9fb-10885801a21f" containerID="badd5120fa261a21202391d4f10736dc2aceca3ee86f9d9428525add2140f4fd" exitCode=0
Oct 30 19:38:42.654095 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:42.653723    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82012b-xc65q" event={"ID":"5c267fa9-6d83-4c9e-a9fb-10885801a21f","Type":"ContainerDied","Data":"badd5120fa261a21202391d4f10736dc2aceca3ee86f9d9428525add2140f4fd"}
Oct 30 19:38:42.655788 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:42.655764    5043 generic.go:356] "Generic (PLEG): container finished" podID="0615f18f-9ed4-4d64-bc4f-1a57e15286cf" containerID="f5dd80eab41e12d10a569be0d78c28bb4ebd90b8ce71880650c50736fc81e279" exitCode=0
Oct 30 19:38:42.655923 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:42.655840    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6" event={"ID":"0615f18f-9ed4-4d64-bc4f-1a57e15286cf","Type":"ContainerDied","Data":"f5dd80eab41e12d10a569be0d78c28bb4ebd90b8ce71880650c50736fc81e279"}
Oct 30 19:38:42.655923 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:42.655890    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6" event={"ID":"0615f18f-9ed4-4d64-bc4f-1a57e15286cf","Type":"ContainerDied","Data":"a51a3728d77832f32bf43e4944c4a303eaf2c186a8e9d13549d8b99153d0e5f1"}
Oct 30 19:38:42.655923 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:42.655906    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a51a3728d77832f32bf43e4944c4a303eaf2c186a8e9d13549d8b99153d0e5f1"
Oct 30 19:38:42.655923 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:42.655920    5043 scope.go:122] "RemoveContainer" containerID="c583cc7aee0b638dc10d53a5b958f084610eede17043d5993b893fe520295561"
Oct 30 19:38:42.677265 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:38:42.677244    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c583cc7aee0b638dc10d53a5b958f084610eede17043d5993b893fe520295561\": container with ID starting with c583cc7aee0b638dc10d53a5b958f084610eede17043d5993b893fe520295561 not found: ID does not exist" containerID="c583cc7aee0b638dc10d53a5b958f084610eede17043d5993b893fe520295561"
Oct 30 19:38:42.681212 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:42.681188    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6"
Oct 30 19:38:42.810635 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:42.810604    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-workload-socket\") pod \"0615f18f-9ed4-4d64-bc4f-1a57e15286cf\" (UID: \"0615f18f-9ed4-4d64-bc4f-1a57e15286cf\") "
Oct 30 19:38:42.810858 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:42.810657    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-istio-token\" (UniqueName: \"kubernetes.io/projected/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-istio-token\") pod \"0615f18f-9ed4-4d64-bc4f-1a57e15286cf\" (UID: \"0615f18f-9ed4-4d64-bc4f-1a57e15286cf\") "
Oct 30 19:38:42.810858 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:42.810798    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-workload-certs\") pod \"0615f18f-9ed4-4d64-bc4f-1a57e15286cf\" (UID: \"0615f18f-9ed4-4d64-bc4f-1a57e15286cf\") "
Oct 30 19:38:42.810858 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:42.810853    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-istio-podinfo\") pod \"0615f18f-9ed4-4d64-bc4f-1a57e15286cf\" (UID: \"0615f18f-9ed4-4d64-bc4f-1a57e15286cf\") "
Oct 30 19:38:42.811044 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:42.810903    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-istio-data\") pod \"0615f18f-9ed4-4d64-bc4f-1a57e15286cf\" (UID: \"0615f18f-9ed4-4d64-bc4f-1a57e15286cf\") "
Oct 30 19:38:42.811044 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:42.810923    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-istio-envoy\") pod \"0615f18f-9ed4-4d64-bc4f-1a57e15286cf\" (UID: \"0615f18f-9ed4-4d64-bc4f-1a57e15286cf\") "
Oct 30 19:38:42.811044 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:42.810976    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-istiod-ca-cert\") pod \"0615f18f-9ed4-4d64-bc4f-1a57e15286cf\" (UID: \"0615f18f-9ed4-4d64-bc4f-1a57e15286cf\") "
Oct 30 19:38:42.811044 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:42.810995    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-workload-socket" pod "0615f18f-9ed4-4d64-bc4f-1a57e15286cf" (UID: "0615f18f-9ed4-4d64-bc4f-1a57e15286cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:42.811044 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:42.811024    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-kube-api-access-f9mgm\" (UniqueName: \"kubernetes.io/projected/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-kube-api-access-f9mgm\") pod \"0615f18f-9ed4-4d64-bc4f-1a57e15286cf\" (UID: \"0615f18f-9ed4-4d64-bc4f-1a57e15286cf\") "
Oct 30 19:38:42.811306 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:42.811082    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-credential-socket\") pod \"0615f18f-9ed4-4d64-bc4f-1a57e15286cf\" (UID: \"0615f18f-9ed4-4d64-bc4f-1a57e15286cf\") "
Oct 30 19:38:42.811306 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:42.811109    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-workload-certs" pod "0615f18f-9ed4-4d64-bc4f-1a57e15286cf" (UID: "0615f18f-9ed4-4d64-bc4f-1a57e15286cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:42.811306 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:42.811158    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-istio-data" pod "0615f18f-9ed4-4d64-bc4f-1a57e15286cf" (UID: "0615f18f-9ed4-4d64-bc4f-1a57e15286cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:42.811459 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:42.811332    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-istiod-ca-cert" pod "0615f18f-9ed4-4d64-bc4f-1a57e15286cf" (UID: "0615f18f-9ed4-4d64-bc4f-1a57e15286cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:38:42.811459 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:42.811422    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-credential-socket" pod "0615f18f-9ed4-4d64-bc4f-1a57e15286cf" (UID: "0615f18f-9ed4-4d64-bc4f-1a57e15286cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:42.812270 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:42.812244    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:42.812406 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:42.812275    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:42.812406 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:42.812290    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:42.812406 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:42.812305    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:42.812406 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:42.812319    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:42.813400 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:42.813372    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-istio-token" pod "0615f18f-9ed4-4d64-bc4f-1a57e15286cf" (UID: "0615f18f-9ed4-4d64-bc4f-1a57e15286cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:42.813400 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:42.813389    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-istio-envoy" pod "0615f18f-9ed4-4d64-bc4f-1a57e15286cf" (UID: "0615f18f-9ed4-4d64-bc4f-1a57e15286cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:42.813518 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:42.813395    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-kube-api-access-f9mgm" pod "0615f18f-9ed4-4d64-bc4f-1a57e15286cf" (UID: "0615f18f-9ed4-4d64-bc4f-1a57e15286cf"). InnerVolumeSpecName "kube-api-access-f9mgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:42.813567 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:42.813552    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-istio-podinfo" pod "0615f18f-9ed4-4d64-bc4f-1a57e15286cf" (UID: "0615f18f-9ed4-4d64-bc4f-1a57e15286cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:38:42.914352 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:42.914321    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:42.914352 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:42.914351    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:42.914524 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:42.914367    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:42.914524 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:42.914376    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9mgm\" (UniqueName: \"kubernetes.io/projected/0615f18f-9ed4-4d64-bc4f-1a57e15286cf-kube-api-access-f9mgm\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:43.282769 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:43.280848    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6a6348-46bdl"]
Oct 30 19:38:43.323764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:43.321111    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6a6348-46bdl"]
Oct 30 19:38:43.668649 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:43.668564    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6"
Oct 30 19:38:43.706433 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:43.706397    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6"]
Oct 30 19:38:43.712548 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:43.712517    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-395ccc-openshift-default-5dcf9cb54c-fnlf6"]
Oct 30 19:38:43.843264 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:43.843238    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82012b-xc65q"
Oct 30 19:38:43.941565 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:43.941455    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c267fa9-6d83-4c9e-a9fb-10885801a21f-kube-api-access-c5f8h\" (UniqueName: \"kubernetes.io/projected/5c267fa9-6d83-4c9e-a9fb-10885801a21f-kube-api-access-c5f8h\") pod \"5c267fa9-6d83-4c9e-a9fb-10885801a21f\" (UID: \"5c267fa9-6d83-4c9e-a9fb-10885801a21f\") "
Oct 30 19:38:43.941784 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:43.941662    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c267fa9-6d83-4c9e-a9fb-10885801a21f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c267fa9-6d83-4c9e-a9fb-10885801a21f-maas-api-openshift-ca\") pod \"5c267fa9-6d83-4c9e-a9fb-10885801a21f\" (UID: \"5c267fa9-6d83-4c9e-a9fb-10885801a21f\") "
Oct 30 19:38:43.941887 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:43.941828    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c267fa9-6d83-4c9e-a9fb-10885801a21f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c267fa9-6d83-4c9e-a9fb-10885801a21f-maas-api-xks-ca\") pod \"5c267fa9-6d83-4c9e-a9fb-10885801a21f\" (UID: \"5c267fa9-6d83-4c9e-a9fb-10885801a21f\") "
Oct 30 19:38:43.942112 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:43.942080    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c267fa9-6d83-4c9e-a9fb-10885801a21f-maas-api-openshift-ca" pod "5c267fa9-6d83-4c9e-a9fb-10885801a21f" (UID: "5c267fa9-6d83-4c9e-a9fb-10885801a21f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:38:43.942503 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:43.942484    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c267fa9-6d83-4c9e-a9fb-10885801a21f-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:43.944112 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:43.944081    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c267fa9-6d83-4c9e-a9fb-10885801a21f-maas-api-xks-ca" pod "5c267fa9-6d83-4c9e-a9fb-10885801a21f" (UID: "5c267fa9-6d83-4c9e-a9fb-10885801a21f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:38:43.944578 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:43.944550    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c267fa9-6d83-4c9e-a9fb-10885801a21f-kube-api-access-c5f8h" pod "5c267fa9-6d83-4c9e-a9fb-10885801a21f" (UID: "5c267fa9-6d83-4c9e-a9fb-10885801a21f"). InnerVolumeSpecName "kube-api-access-c5f8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:44.044305 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:44.044276    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c267fa9-6d83-4c9e-a9fb-10885801a21f-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:44.044305 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:44.044304    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5f8h\" (UniqueName: \"kubernetes.io/projected/5c267fa9-6d83-4c9e-a9fb-10885801a21f-kube-api-access-c5f8h\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:44.528631 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:44.528604    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1e13df-78b7ccf4bc-8ldn2"
Oct 30 19:38:44.653915 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:44.653884    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0615f18f-9ed4-4d64-bc4f-1a57e15286cf" path="/var/lib/kubelet/pods/0615f18f-9ed4-4d64-bc4f-1a57e15286cf/volumes"
Oct 30 19:38:44.654900 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:44.654838    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="869b0618-5974-4bc0-91cf-339f483d115f" path="/var/lib/kubelet/pods/869b0618-5974-4bc0-91cf-339f483d115f/volumes"
Oct 30 19:38:44.682086 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:44.682058    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82012b-xc65q"
Oct 30 19:38:44.682548 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:44.682104    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82012b-xc65q" event={"ID":"5c267fa9-6d83-4c9e-a9fb-10885801a21f","Type":"ContainerDied","Data":"8bf4aec55bd1f5459eaf782f5f7113dd20192608d1c091f642f376c378a6a9c8"}
Oct 30 19:38:44.682548 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:44.682159    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bf4aec55bd1f5459eaf782f5f7113dd20192608d1c091f642f376c378a6a9c8"
Oct 30 19:38:47.553391 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:47.553352    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-22af66-7b5b988746-mm9zl"]
Oct 30 19:38:47.554212 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:47.554194    5043 cpu_manager.go:414] "containerMap: removing container" podUID="5c267fa9-6d83-4c9e-a9fb-10885801a21f" containerName="revoke-keys"
Oct 30 19:38:47.554313 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:47.554214    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c267fa9-6d83-4c9e-a9fb-10885801a21f" containerName="revoke-keys"
Oct 30 19:38:47.554313 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:47.554225    5043 cpu_manager.go:414] "containerMap: removing container" podUID="0615f18f-9ed4-4d64-bc4f-1a57e15286cf" containerName="istio-proxy"
Oct 30 19:38:47.554313 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:47.554230    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0615f18f-9ed4-4d64-bc4f-1a57e15286cf" containerName="istio-proxy"
Oct 30 19:38:47.554313 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:47.554272    5043 cpu_manager.go:414] "containerMap: removing container" podUID="346d6503-7d76-47c4-8528-18d1f0669680" containerName="revoke-keys"
Oct 30 19:38:47.554313 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:47.554278    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="346d6503-7d76-47c4-8528-18d1f0669680" containerName="revoke-keys"
Oct 30 19:38:47.554485 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:47.554390    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="346d6503-7d76-47c4-8528-18d1f0669680" containerName="revoke-keys"
Oct 30 19:38:47.554485 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:47.554399    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="0615f18f-9ed4-4d64-bc4f-1a57e15286cf" containerName="istio-proxy"
Oct 30 19:38:47.554485 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:47.554409    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c267fa9-6d83-4c9e-a9fb-10885801a21f" containerName="revoke-keys"
Oct 30 19:38:47.559594 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:47.559573    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-22af66-7b5b988746-mm9zl"
Oct 30 19:38:47.567998 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:47.567974    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-22af66\""
Oct 30 19:38:47.571785 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:47.569831    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-22af66-dockercfg-jnw5m\""
Oct 30 19:38:47.582046 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:47.582019    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-22af66-7b5b988746-mm9zl"]
Oct 30 19:38:47.605844 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:47.605811    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-22af66-5bf7f46cd7-ffx8p"]
Oct 30 19:38:47.606546 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:47.606530    5043 cpu_manager.go:414] "containerMap: removing container" podUID="0615f18f-9ed4-4d64-bc4f-1a57e15286cf" containerName="istio-proxy"
Oct 30 19:38:47.606546 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:47.606547    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0615f18f-9ed4-4d64-bc4f-1a57e15286cf" containerName="istio-proxy"
Oct 30 19:38:47.606708 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:47.606698    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="0615f18f-9ed4-4d64-bc4f-1a57e15286cf" containerName="istio-proxy"
Oct 30 19:38:47.611446 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:47.611267    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-22af66-5bf7f46cd7-ffx8p"
Oct 30 19:38:47.615779 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:47.614627    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-22af66-5bf7f46cd7-ffx8p"]
Oct 30 19:38:47.969139 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:47.963610    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/183c301b-de2c-4f03-b142-d53819964452-plugins-config-volume\") pod \"payload-processing-e2e-api-b-22af66-7b5b988746-mm9zl\" (UID: \"183c301b-de2c-4f03-b142-d53819964452\") " pod="openshift-ingress/payload-processing-e2e-api-b-22af66-7b5b988746-mm9zl"
Oct 30 19:38:47.969139 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:47.963996    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkd7z\" (UniqueName: \"kubernetes.io/projected/183c301b-de2c-4f03-b142-d53819964452-kube-api-access-gkd7z\") pod \"payload-processing-e2e-api-b-22af66-7b5b988746-mm9zl\" (UID: \"183c301b-de2c-4f03-b142-d53819964452\") " pod="openshift-ingress/payload-processing-e2e-api-b-22af66-7b5b988746-mm9zl"
Oct 30 19:38:47.969139 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:47.964301    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c12af95-b3fd-4146-857b-27c42aa4dc74-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-22af66-5bf7f46cd7-ffx8p\" (UID: \"8c12af95-b3fd-4146-857b-27c42aa4dc74\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-22af66-5bf7f46cd7-ffx8p"
Oct 30 19:38:47.969139 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:47.964369    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grsjj\" (UniqueName: \"kubernetes.io/projected/8c12af95-b3fd-4146-857b-27c42aa4dc74-kube-api-access-grsjj\") pod \"payload-pre-processing-e2e-api-b-22af66-5bf7f46cd7-ffx8p\" (UID: \"8c12af95-b3fd-4146-857b-27c42aa4dc74\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-22af66-5bf7f46cd7-ffx8p"
Oct 30 19:38:47.969139 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:47.965544    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-ae425f-5885c86dd4-4wvh9"]
Oct 30 19:38:47.969139 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:47.967847    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ae425f-79c9ff7db7-z7nf7"]
Oct 30 19:38:48.065460 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:48.065425    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkd7z\" (UniqueName: \"kubernetes.io/projected/183c301b-de2c-4f03-b142-d53819964452-kube-api-access-gkd7z\") pod \"payload-processing-e2e-api-b-22af66-7b5b988746-mm9zl\" (UID: \"183c301b-de2c-4f03-b142-d53819964452\") " pod="openshift-ingress/payload-processing-e2e-api-b-22af66-7b5b988746-mm9zl"
Oct 30 19:38:48.065666 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:48.065555    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c12af95-b3fd-4146-857b-27c42aa4dc74-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-22af66-5bf7f46cd7-ffx8p\" (UID: \"8c12af95-b3fd-4146-857b-27c42aa4dc74\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-22af66-5bf7f46cd7-ffx8p"
Oct 30 19:38:48.065666 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:48.065593    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grsjj\" (UniqueName: \"kubernetes.io/projected/8c12af95-b3fd-4146-857b-27c42aa4dc74-kube-api-access-grsjj\") pod \"payload-pre-processing-e2e-api-b-22af66-5bf7f46cd7-ffx8p\" (UID: \"8c12af95-b3fd-4146-857b-27c42aa4dc74\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-22af66-5bf7f46cd7-ffx8p"
Oct 30 19:38:48.066636 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:48.065694    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/183c301b-de2c-4f03-b142-d53819964452-plugins-config-volume\") pod \"payload-processing-e2e-api-b-22af66-7b5b988746-mm9zl\" (UID: \"183c301b-de2c-4f03-b142-d53819964452\") " pod="openshift-ingress/payload-processing-e2e-api-b-22af66-7b5b988746-mm9zl"
Oct 30 19:38:48.066636 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:48.066401    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c12af95-b3fd-4146-857b-27c42aa4dc74-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-22af66-5bf7f46cd7-ffx8p\" (UID: \"8c12af95-b3fd-4146-857b-27c42aa4dc74\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-22af66-5bf7f46cd7-ffx8p"
Oct 30 19:38:48.066636 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:48.066449    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/183c301b-de2c-4f03-b142-d53819964452-plugins-config-volume\") pod \"payload-processing-e2e-api-b-22af66-7b5b988746-mm9zl\" (UID: \"183c301b-de2c-4f03-b142-d53819964452\") " pod="openshift-ingress/payload-processing-e2e-api-b-22af66-7b5b988746-mm9zl"
Oct 30 19:38:48.074589 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:48.074567    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkd7z\" (UniqueName: \"kubernetes.io/projected/183c301b-de2c-4f03-b142-d53819964452-kube-api-access-gkd7z\") pod \"payload-processing-e2e-api-b-22af66-7b5b988746-mm9zl\" (UID: \"183c301b-de2c-4f03-b142-d53819964452\") " pod="openshift-ingress/payload-processing-e2e-api-b-22af66-7b5b988746-mm9zl"
Oct 30 19:38:48.075494 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:48.075469    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grsjj\" (UniqueName: \"kubernetes.io/projected/8c12af95-b3fd-4146-857b-27c42aa4dc74-kube-api-access-grsjj\") pod \"payload-pre-processing-e2e-api-b-22af66-5bf7f46cd7-ffx8p\" (UID: \"8c12af95-b3fd-4146-857b-27c42aa4dc74\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-22af66-5bf7f46cd7-ffx8p"
Oct 30 19:38:48.200760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:48.189458    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-22af66-7b5b988746-mm9zl"
Oct 30 19:38:48.278143 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:48.278062    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-22af66-5bf7f46cd7-ffx8p"
Oct 30 19:38:48.774529 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:48.774501    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-22af66-5bf7f46cd7-ffx8p"]
Oct 30 19:38:48.775286 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:38:48.774890    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod183c301b_de2c_4f03_b142_d53819964452.slice/crio-a0627b5aefbc93810c4eefd6756791b974437f70d550f4b900a239182ef4a967 WatchSource:0}: Error finding container a0627b5aefbc93810c4eefd6756791b974437f70d550f4b900a239182ef4a967: Status 404 returned error can't find the container with id a0627b5aefbc93810c4eefd6756791b974437f70d550f4b900a239182ef4a967
Oct 30 19:38:48.780437 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:48.780382    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-22af66-7b5b988746-mm9zl"]
Oct 30 19:38:48.975057 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:48.974882    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-22af66-5bf7f46cd7-ffx8p" event={"ID":"8c12af95-b3fd-4146-857b-27c42aa4dc74","Type":"ContainerStarted","Data":"54ca03c314cafb968d3574a7afda707d1baa18147ee1390af40ac827affc31d0"}
Oct 30 19:38:48.975057 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:48.974960    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-22af66-5bf7f46cd7-ffx8p" event={"ID":"8c12af95-b3fd-4146-857b-27c42aa4dc74","Type":"ContainerStarted","Data":"19cb30b379fd8815c00a221cdc4473214d174fe0ceace125107d5a7f9e992521"}
Oct 30 19:38:48.975057 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:48.975014    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-22af66-5bf7f46cd7-ffx8p"
Oct 30 19:38:48.977145 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:48.977123    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-22af66-7b5b988746-mm9zl" event={"ID":"183c301b-de2c-4f03-b142-d53819964452","Type":"ContainerStarted","Data":"a0627b5aefbc93810c4eefd6756791b974437f70d550f4b900a239182ef4a967"}
Oct 30 19:38:48.997655 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:48.997346    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-22af66-5bf7f46cd7-ffx8p" podStartSLOduration=1.997332706 podStartE2EDuration="1.997332706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:48.993484641 +0000 UTC m=+1348.504544607" watchObservedRunningTime="2026-10-30 19:38:48.997332706 +0000 UTC m=+1348.508392671"
Oct 30 19:38:49.997483 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:49.997447    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-22af66-7b5b988746-mm9zl" event={"ID":"183c301b-de2c-4f03-b142-d53819964452","Type":"ContainerStarted","Data":"3337b9dfef89f070dd0a168af76f962595f790f202464508cb16ee6dbfd653be"}
Oct 30 19:38:49.998281 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:49.998245    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-22af66-7b5b988746-mm9zl"
Oct 30 19:38:50.021415 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:50.021349    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-22af66-7b5b988746-mm9zl" podStartSLOduration=3.021330197 podStartE2EDuration="3.021330197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:50.014292623 +0000 UTC m=+1349.525352569" watchObservedRunningTime="2026-10-30 19:38:50.021330197 +0000 UTC m=+1349.532390158"
Oct 30 19:38:51.016041 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:51.016012    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-82012b-5987f89c59-xqtgs"]
Oct 30 19:38:51.046189 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:51.046152    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-673809b4-ps722"]
Oct 30 19:38:51.069776 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:51.061920    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-673809b4-ps722"]
Oct 30 19:38:51.719028 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:51.718990    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5"]
Oct 30 19:38:51.719292 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:51.719259    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5" podUID="aa769f06-2fc2-40ce-bb1a-a198b2274afb" containerName="istio-proxy" containerID="cri-o://3766a7f3ae3915d6bb7b92255e03f826433e7da7db7ec6e8be797782bd995437" gracePeriod=30
Oct 30 19:38:52.649965 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:52.649929    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="024272f8-924a-4157-a1a5-4c1ebc5ec4b9" path="/var/lib/kubelet/pods/024272f8-924a-4157-a1a5-4c1ebc5ec4b9/volumes"
Oct 30 19:38:52.927221 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:52.927138    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd"]
Oct 30 19:38:52.934624 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:52.934593    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd"
Oct 30 19:38:52.948775 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:52.945035    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-744b57-openshift-default-dockercfg-xvhf2\""
Oct 30 19:38:52.951665 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:52.951631    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd"]
Oct 30 19:38:52.970572 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:52.970476    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ae425f-79c9ff7db7-z7nf7" podUID="7753bd1c-f916-40db-a923-eb46e92a548e" containerName="payload-pre-processing" containerID="cri-o://c32fff151da4aebefbb4b03edc9ac18a11158116f76425a2078c0c6f4c1d4174" gracePeriod=25
Oct 30 19:38:52.974387 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:52.974219    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ae425f-5885c86dd4-4wvh9" podUID="469e981b-1193-4d89-b98a-bc6f794cb532" containerName="payload-processing" containerID="cri-o://92823e6406886c547be12b046c1d35705116dc334827abd330b1831e7563d89c" gracePeriod=25
Oct 30 19:38:53.010042 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:38:53.004616    5043 logging.go:55] [core] [Channel #2241 SubChannel #2243]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.159:50052", ServerName: "10.132.0.159:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.159:50052: connect: connection refused"
Oct 30 19:38:53.010414 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:38:53.010395    5043 logging.go:55] [core] [Channel #2240 SubChannel #2242]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.160:50052", ServerName: "10.132.0.160:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.160:50052: connect: connection refused"
Oct 30 19:38:53.127778 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:53.119713    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ff1923b-6547-4e9b-8810-3f79648e3272-istio-envoy\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd\" (UID: \"5ff1923b-6547-4e9b-8810-3f79648e3272\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd"
Oct 30 19:38:53.127778 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:53.119842    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ff1923b-6547-4e9b-8810-3f79648e3272-workload-certs\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd\" (UID: \"5ff1923b-6547-4e9b-8810-3f79648e3272\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd"
Oct 30 19:38:53.127778 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:53.119879    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ff1923b-6547-4e9b-8810-3f79648e3272-istio-data\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd\" (UID: \"5ff1923b-6547-4e9b-8810-3f79648e3272\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd"
Oct 30 19:38:53.127778 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:53.119906    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ff1923b-6547-4e9b-8810-3f79648e3272-istio-token\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd\" (UID: \"5ff1923b-6547-4e9b-8810-3f79648e3272\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd"
Oct 30 19:38:53.127778 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:53.120000    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ff1923b-6547-4e9b-8810-3f79648e3272-istiod-ca-cert\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd\" (UID: \"5ff1923b-6547-4e9b-8810-3f79648e3272\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd"
Oct 30 19:38:53.127778 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:53.120071    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ff1923b-6547-4e9b-8810-3f79648e3272-workload-socket\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd\" (UID: \"5ff1923b-6547-4e9b-8810-3f79648e3272\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd"
Oct 30 19:38:53.127778 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:53.120114    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ff1923b-6547-4e9b-8810-3f79648e3272-credential-socket\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd\" (UID: \"5ff1923b-6547-4e9b-8810-3f79648e3272\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd"
Oct 30 19:38:53.127778 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:53.120172    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd494\" (UniqueName: \"kubernetes.io/projected/5ff1923b-6547-4e9b-8810-3f79648e3272-kube-api-access-qd494\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd\" (UID: \"5ff1923b-6547-4e9b-8810-3f79648e3272\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd"
Oct 30 19:38:53.127778 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:53.120209    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ff1923b-6547-4e9b-8810-3f79648e3272-istio-podinfo\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd\" (UID: \"5ff1923b-6547-4e9b-8810-3f79648e3272\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd"
Oct 30 19:38:53.278774 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:53.268779    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ff1923b-6547-4e9b-8810-3f79648e3272-workload-socket\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd\" (UID: \"5ff1923b-6547-4e9b-8810-3f79648e3272\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd"
Oct 30 19:38:53.278774 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:53.268855    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ff1923b-6547-4e9b-8810-3f79648e3272-credential-socket\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd\" (UID: \"5ff1923b-6547-4e9b-8810-3f79648e3272\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd"
Oct 30 19:38:53.278774 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:53.268916    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd494\" (UniqueName: \"kubernetes.io/projected/5ff1923b-6547-4e9b-8810-3f79648e3272-kube-api-access-qd494\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd\" (UID: \"5ff1923b-6547-4e9b-8810-3f79648e3272\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd"
Oct 30 19:38:53.278774 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:53.268961    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ff1923b-6547-4e9b-8810-3f79648e3272-istio-podinfo\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd\" (UID: \"5ff1923b-6547-4e9b-8810-3f79648e3272\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd"
Oct 30 19:38:53.278774 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:53.269153    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ff1923b-6547-4e9b-8810-3f79648e3272-istio-envoy\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd\" (UID: \"5ff1923b-6547-4e9b-8810-3f79648e3272\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd"
Oct 30 19:38:53.278774 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:53.269226    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ff1923b-6547-4e9b-8810-3f79648e3272-workload-certs\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd\" (UID: \"5ff1923b-6547-4e9b-8810-3f79648e3272\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd"
Oct 30 19:38:53.278774 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:53.269259    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ff1923b-6547-4e9b-8810-3f79648e3272-istio-data\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd\" (UID: \"5ff1923b-6547-4e9b-8810-3f79648e3272\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd"
Oct 30 19:38:53.278774 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:53.269286    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ff1923b-6547-4e9b-8810-3f79648e3272-istio-token\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd\" (UID: \"5ff1923b-6547-4e9b-8810-3f79648e3272\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd"
Oct 30 19:38:53.278774 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:53.269373    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ff1923b-6547-4e9b-8810-3f79648e3272-istiod-ca-cert\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd\" (UID: \"5ff1923b-6547-4e9b-8810-3f79648e3272\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd"
Oct 30 19:38:53.297814 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:53.289926    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ff1923b-6547-4e9b-8810-3f79648e3272-workload-socket\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd\" (UID: \"5ff1923b-6547-4e9b-8810-3f79648e3272\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd"
Oct 30 19:38:53.297814 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:53.292241    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ff1923b-6547-4e9b-8810-3f79648e3272-istio-podinfo\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd\" (UID: \"5ff1923b-6547-4e9b-8810-3f79648e3272\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd"
Oct 30 19:38:53.297814 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:53.292494    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ff1923b-6547-4e9b-8810-3f79648e3272-credential-socket\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd\" (UID: \"5ff1923b-6547-4e9b-8810-3f79648e3272\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd"
Oct 30 19:38:53.297814 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:53.293068    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ff1923b-6547-4e9b-8810-3f79648e3272-istio-data\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd\" (UID: \"5ff1923b-6547-4e9b-8810-3f79648e3272\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd"
Oct 30 19:38:53.299262 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:53.299182    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ff1923b-6547-4e9b-8810-3f79648e3272-istiod-ca-cert\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd\" (UID: \"5ff1923b-6547-4e9b-8810-3f79648e3272\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd"
Oct 30 19:38:53.299681 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:53.299635    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ff1923b-6547-4e9b-8810-3f79648e3272-workload-certs\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd\" (UID: \"5ff1923b-6547-4e9b-8810-3f79648e3272\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd"
Oct 30 19:38:53.340025 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:53.339972    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd494\" (UniqueName: \"kubernetes.io/projected/5ff1923b-6547-4e9b-8810-3f79648e3272-kube-api-access-qd494\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd\" (UID: \"5ff1923b-6547-4e9b-8810-3f79648e3272\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd"
Oct 30 19:38:53.346410 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:53.346377    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ff1923b-6547-4e9b-8810-3f79648e3272-istio-envoy\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd\" (UID: \"5ff1923b-6547-4e9b-8810-3f79648e3272\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd"
Oct 30 19:38:53.349842 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:53.349486    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ff1923b-6547-4e9b-8810-3f79648e3272-istio-token\") pod \"e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd\" (UID: \"5ff1923b-6547-4e9b-8810-3f79648e3272\") " pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd"
Oct 30 19:38:53.553397 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:53.553016    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ae425f-79c9ff7db7-z7nf7"
Oct 30 19:38:53.564587 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:53.564552    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd"
Oct 30 19:38:53.677773 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:53.676595    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ae425f-5885c86dd4-4wvh9"
Oct 30 19:38:53.694877 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:53.694693    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7753bd1c-f916-40db-a923-eb46e92a548e-kube-api-access-q5n7w\" (UniqueName: \"kubernetes.io/projected/7753bd1c-f916-40db-a923-eb46e92a548e-kube-api-access-q5n7w\") pod \"7753bd1c-f916-40db-a923-eb46e92a548e\" (UID: \"7753bd1c-f916-40db-a923-eb46e92a548e\") "
Oct 30 19:38:53.695422 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:53.694976    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7753bd1c-f916-40db-a923-eb46e92a548e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7753bd1c-f916-40db-a923-eb46e92a548e-plugins-config-volume\") pod \"7753bd1c-f916-40db-a923-eb46e92a548e\" (UID: \"7753bd1c-f916-40db-a923-eb46e92a548e\") "
Oct 30 19:38:53.696852 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:53.695809    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7753bd1c-f916-40db-a923-eb46e92a548e-plugins-config-volume" pod "7753bd1c-f916-40db-a923-eb46e92a548e" (UID: "7753bd1c-f916-40db-a923-eb46e92a548e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:38:53.699811 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:53.699759    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7753bd1c-f916-40db-a923-eb46e92a548e-kube-api-access-q5n7w" pod "7753bd1c-f916-40db-a923-eb46e92a548e" (UID: "7753bd1c-f916-40db-a923-eb46e92a548e"). InnerVolumeSpecName "kube-api-access-q5n7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:53.805812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:53.801338    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/469e981b-1193-4d89-b98a-bc6f794cb532-kube-api-access-4h86w\" (UniqueName: \"kubernetes.io/projected/469e981b-1193-4d89-b98a-bc6f794cb532-kube-api-access-4h86w\") pod \"469e981b-1193-4d89-b98a-bc6f794cb532\" (UID: \"469e981b-1193-4d89-b98a-bc6f794cb532\") "
Oct 30 19:38:53.805812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:53.801557    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/469e981b-1193-4d89-b98a-bc6f794cb532-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/469e981b-1193-4d89-b98a-bc6f794cb532-plugins-config-volume\") pod \"469e981b-1193-4d89-b98a-bc6f794cb532\" (UID: \"469e981b-1193-4d89-b98a-bc6f794cb532\") "
Oct 30 19:38:53.805812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:53.802517    5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7753bd1c-f916-40db-a923-eb46e92a548e-plugins-config-volume\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:53.805812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:53.802545    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5n7w\" (UniqueName: \"kubernetes.io/projected/7753bd1c-f916-40db-a923-eb46e92a548e-kube-api-access-q5n7w\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:53.849052 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:53.848995    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/469e981b-1193-4d89-b98a-bc6f794cb532-kube-api-access-4h86w" pod "469e981b-1193-4d89-b98a-bc6f794cb532" (UID: "469e981b-1193-4d89-b98a-bc6f794cb532"). InnerVolumeSpecName "kube-api-access-4h86w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:53.876159 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:53.876083    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/469e981b-1193-4d89-b98a-bc6f794cb532-plugins-config-volume" pod "469e981b-1193-4d89-b98a-bc6f794cb532" (UID: "469e981b-1193-4d89-b98a-bc6f794cb532"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:38:53.906386 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:53.906235    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h86w\" (UniqueName: \"kubernetes.io/projected/469e981b-1193-4d89-b98a-bc6f794cb532-kube-api-access-4h86w\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:53.906386 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:53.906267    5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/469e981b-1193-4d89-b98a-bc6f794cb532-plugins-config-volume\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:53.962535 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:38:53.962498    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ff1923b_6547_4e9b_8810_3f79648e3272.slice/crio-0ee9c1430b6ccd0b2c31af897216adc76d3932093a50a28f3c5d83a0fbbb01a4 WatchSource:0}: Error finding container 0ee9c1430b6ccd0b2c31af897216adc76d3932093a50a28f3c5d83a0fbbb01a4: Status 404 returned error can't find the container with id 0ee9c1430b6ccd0b2c31af897216adc76d3932093a50a28f3c5d83a0fbbb01a4
Oct 30 19:38:53.966835 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:53.966185    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:38:53.966835 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:53.966279    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:38:53.966835 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:53.966324    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:38:53.988636 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:53.975400    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd"]
Oct 30 19:38:54.008247 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:54.004964    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ae425f-5885c86dd4-4wvh9" podUID="469e981b-1193-4d89-b98a-bc6f794cb532" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.160:50052\" within 1s: context deadline exceeded"
Oct 30 19:38:54.008247 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:54.005416    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ae425f-79c9ff7db7-z7nf7" podUID="7753bd1c-f916-40db-a923-eb46e92a548e" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.159:50052\" within 1s: context deadline exceeded"
Oct 30 19:38:54.332801 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:54.332554    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd" event={"ID":"5ff1923b-6547-4e9b-8810-3f79648e3272","Type":"ContainerStarted","Data":"7ec243dad6c669aa3939aa3a8751708fa5f41444c4da0a374873dc8b9e1e3f16"}
Oct 30 19:38:54.332801 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:54.332639    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd" event={"ID":"5ff1923b-6547-4e9b-8810-3f79648e3272","Type":"ContainerStarted","Data":"0ee9c1430b6ccd0b2c31af897216adc76d3932093a50a28f3c5d83a0fbbb01a4"}
Oct 30 19:38:54.334766 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:54.334718    5043 generic.go:356] "Generic (PLEG): container finished" podID="7753bd1c-f916-40db-a923-eb46e92a548e" containerID="c32fff151da4aebefbb4b03edc9ac18a11158116f76425a2078c0c6f4c1d4174" exitCode=0
Oct 30 19:38:54.334909 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:54.334775    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ae425f-79c9ff7db7-z7nf7" event={"ID":"7753bd1c-f916-40db-a923-eb46e92a548e","Type":"ContainerDied","Data":"c32fff151da4aebefbb4b03edc9ac18a11158116f76425a2078c0c6f4c1d4174"}
Oct 30 19:38:54.334909 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:54.334808    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ae425f-79c9ff7db7-z7nf7"
Oct 30 19:38:54.334909 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:54.334845    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ae425f-79c9ff7db7-z7nf7" event={"ID":"7753bd1c-f916-40db-a923-eb46e92a548e","Type":"ContainerDied","Data":"d6de27d69c4218f657a24814d652c79dd780e72cca15159ef63314e07464756a"}
Oct 30 19:38:54.334909 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:54.334877    5043 scope.go:122] "RemoveContainer" containerID="c32fff151da4aebefbb4b03edc9ac18a11158116f76425a2078c0c6f4c1d4174"
Oct 30 19:38:54.337428 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:54.337394    5043 generic.go:356] "Generic (PLEG): container finished" podID="469e981b-1193-4d89-b98a-bc6f794cb532" containerID="92823e6406886c547be12b046c1d35705116dc334827abd330b1831e7563d89c" exitCode=0
Oct 30 19:38:54.337630 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:54.337611    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ae425f-5885c86dd4-4wvh9" event={"ID":"469e981b-1193-4d89-b98a-bc6f794cb532","Type":"ContainerDied","Data":"92823e6406886c547be12b046c1d35705116dc334827abd330b1831e7563d89c"}
Oct 30 19:38:54.337722 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:54.337645    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ae425f-5885c86dd4-4wvh9"
Oct 30 19:38:54.337722 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:54.337660    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ae425f-5885c86dd4-4wvh9" event={"ID":"469e981b-1193-4d89-b98a-bc6f794cb532","Type":"ContainerDied","Data":"7bd85e8380caa5a3b3e981fe3eddd8a500b32f9d1512f84e6cc6cab8e4ad97d2"}
Oct 30 19:38:54.357999 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:54.357967    5043 scope.go:122] "RemoveContainer" containerID="c32fff151da4aebefbb4b03edc9ac18a11158116f76425a2078c0c6f4c1d4174"
Oct 30 19:38:54.358345 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:38:54.358311    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c32fff151da4aebefbb4b03edc9ac18a11158116f76425a2078c0c6f4c1d4174\": container with ID starting with c32fff151da4aebefbb4b03edc9ac18a11158116f76425a2078c0c6f4c1d4174 not found: ID does not exist" containerID="c32fff151da4aebefbb4b03edc9ac18a11158116f76425a2078c0c6f4c1d4174"
Oct 30 19:38:54.358463 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:54.358359    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c32fff151da4aebefbb4b03edc9ac18a11158116f76425a2078c0c6f4c1d4174"} err="failed to get container status \"c32fff151da4aebefbb4b03edc9ac18a11158116f76425a2078c0c6f4c1d4174\": rpc error: code = NotFound desc = could not find container \"c32fff151da4aebefbb4b03edc9ac18a11158116f76425a2078c0c6f4c1d4174\": container with ID starting with c32fff151da4aebefbb4b03edc9ac18a11158116f76425a2078c0c6f4c1d4174 not found: ID does not exist"
Oct 30 19:38:54.358463 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:54.358382    5043 scope.go:122] "RemoveContainer" containerID="92823e6406886c547be12b046c1d35705116dc334827abd330b1831e7563d89c"
Oct 30 19:38:54.359430 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:54.359377    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd" podStartSLOduration=2.359361528 podStartE2EDuration="2.359361528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:54.35619265 +0000 UTC m=+1353.867252618" watchObservedRunningTime="2026-10-30 19:38:54.359361528 +0000 UTC m=+1353.870421493"
Oct 30 19:38:54.378029 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:54.377989    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ae425f-79c9ff7db7-z7nf7"]
Oct 30 19:38:54.381703 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:54.381684    5043 scope.go:122] "RemoveContainer" containerID="92823e6406886c547be12b046c1d35705116dc334827abd330b1831e7563d89c"
Oct 30 19:38:54.382076 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:38:54.382053    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92823e6406886c547be12b046c1d35705116dc334827abd330b1831e7563d89c\": container with ID starting with 92823e6406886c547be12b046c1d35705116dc334827abd330b1831e7563d89c not found: ID does not exist" containerID="92823e6406886c547be12b046c1d35705116dc334827abd330b1831e7563d89c"
Oct 30 19:38:54.382212 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:54.382183    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92823e6406886c547be12b046c1d35705116dc334827abd330b1831e7563d89c"} err="failed to get container status \"92823e6406886c547be12b046c1d35705116dc334827abd330b1831e7563d89c\": rpc error: code = NotFound desc = could not find container \"92823e6406886c547be12b046c1d35705116dc334827abd330b1831e7563d89c\": container with ID starting with 92823e6406886c547be12b046c1d35705116dc334827abd330b1831e7563d89c not found: ID does not exist"
Oct 30 19:38:54.394780 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:54.388242    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ae425f-79c9ff7db7-z7nf7"]
Oct 30 19:38:54.411197 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:54.411167    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-ae425f-5885c86dd4-4wvh9"]
Oct 30 19:38:54.423576 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:54.423551    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-ae425f-5885c86dd4-4wvh9"]
Oct 30 19:38:54.568968 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:54.568931    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd"
Oct 30 19:38:54.686595 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:54.686510    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="469e981b-1193-4d89-b98a-bc6f794cb532" path="/var/lib/kubelet/pods/469e981b-1193-4d89-b98a-bc6f794cb532/volumes"
Oct 30 19:38:54.687816 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:54.687796    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7753bd1c-f916-40db-a923-eb46e92a548e" path="/var/lib/kubelet/pods/7753bd1c-f916-40db-a923-eb46e92a548e/volumes"
Oct 30 19:38:54.980960 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:38:54.980920    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:38:55.590091 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:55.590063    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd"
Oct 30 19:38:56.020523 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:56.020483    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-82012b-5987f89c59-xqtgs" podUID="d41c7166-ac32-4778-8af3-3d32819427e3" containerName="payload-pre-processing" containerID="cri-o://f365854e03ba3afe0c420f7d98e2e5d8ecb4a46404ddb5cac5016a8597c939e1" gracePeriod=25
Oct 30 19:38:56.439365 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:56.439308    5043 generic.go:356] "Generic (PLEG): container finished" podID="d41c7166-ac32-4778-8af3-3d32819427e3" containerID="f365854e03ba3afe0c420f7d98e2e5d8ecb4a46404ddb5cac5016a8597c939e1" exitCode=0
Oct 30 19:38:56.439625 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:56.439598    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-82012b-5987f89c59-xqtgs" event={"ID":"d41c7166-ac32-4778-8af3-3d32819427e3","Type":"ContainerDied","Data":"f365854e03ba3afe0c420f7d98e2e5d8ecb4a46404ddb5cac5016a8597c939e1"}
Oct 30 19:38:56.439973 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:56.439953    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd"
Oct 30 19:38:56.445027 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:56.444683    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd"
Oct 30 19:38:56.735471 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:56.735407    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-82012b-5987f89c59-xqtgs"
Oct 30 19:38:56.747672 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:56.747643    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d41c7166-ac32-4778-8af3-3d32819427e3-kube-api-access-7jng5\" (UniqueName: \"kubernetes.io/projected/d41c7166-ac32-4778-8af3-3d32819427e3-kube-api-access-7jng5\") pod \"d41c7166-ac32-4778-8af3-3d32819427e3\" (UID: \"d41c7166-ac32-4778-8af3-3d32819427e3\") "
Oct 30 19:38:56.747870 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:56.747804    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d41c7166-ac32-4778-8af3-3d32819427e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d41c7166-ac32-4778-8af3-3d32819427e3-plugins-config-volume\") pod \"d41c7166-ac32-4778-8af3-3d32819427e3\" (UID: \"d41c7166-ac32-4778-8af3-3d32819427e3\") "
Oct 30 19:38:56.751395 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:56.751373    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d41c7166-ac32-4778-8af3-3d32819427e3-plugins-config-volume" pod "d41c7166-ac32-4778-8af3-3d32819427e3" (UID: "d41c7166-ac32-4778-8af3-3d32819427e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:38:56.836084 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:56.836044    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d41c7166-ac32-4778-8af3-3d32819427e3-kube-api-access-7jng5" pod "d41c7166-ac32-4778-8af3-3d32819427e3" (UID: "d41c7166-ac32-4778-8af3-3d32819427e3"). InnerVolumeSpecName "kube-api-access-7jng5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:56.852786 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:56.851676    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jng5\" (UniqueName: \"kubernetes.io/projected/d41c7166-ac32-4778-8af3-3d32819427e3-kube-api-access-7jng5\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:56.852786 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:56.851712    5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d41c7166-ac32-4778-8af3-3d32819427e3-plugins-config-volume\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:57.597761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:57.596915    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-82012b-5987f89c59-xqtgs" event={"ID":"d41c7166-ac32-4778-8af3-3d32819427e3","Type":"ContainerDied","Data":"7deef02642f4c1d933653e90d1ec0bb2656764db4b703e604b6fc19ca852ff98"}
Oct 30 19:38:57.597761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:57.596988    5043 scope.go:122] "RemoveContainer" containerID="f365854e03ba3afe0c420f7d98e2e5d8ecb4a46404ddb5cac5016a8597c939e1"
Oct 30 19:38:57.597761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:57.597202    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-82012b-5987f89c59-xqtgs"
Oct 30 19:38:57.637812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:57.612656    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5"
Oct 30 19:38:57.637812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:57.614081    5043 generic.go:356] "Generic (PLEG): container finished" podID="aa769f06-2fc2-40ce-bb1a-a198b2274afb" containerID="3766a7f3ae3915d6bb7b92255e03f826433e7da7db7ec6e8be797782bd995437" exitCode=0
Oct 30 19:38:57.637812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:57.616792    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5" event={"ID":"aa769f06-2fc2-40ce-bb1a-a198b2274afb","Type":"ContainerDied","Data":"3766a7f3ae3915d6bb7b92255e03f826433e7da7db7ec6e8be797782bd995437"}
Oct 30 19:38:57.637812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:57.616846    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5" event={"ID":"aa769f06-2fc2-40ce-bb1a-a198b2274afb","Type":"ContainerDied","Data":"ac9ee860842ed64719bf87ec53dcee050d8a4c40e3668e3e607a9887ff0d4563"}
Oct 30 19:38:57.663221 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:57.660950    5043 scope.go:122] "RemoveContainer" containerID="3766a7f3ae3915d6bb7b92255e03f826433e7da7db7ec6e8be797782bd995437"
Oct 30 19:38:57.747166 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:57.746776    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa769f06-2fc2-40ce-bb1a-a198b2274afb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa769f06-2fc2-40ce-bb1a-a198b2274afb-istio-envoy\") pod \"aa769f06-2fc2-40ce-bb1a-a198b2274afb\" (UID: \"aa769f06-2fc2-40ce-bb1a-a198b2274afb\") "
Oct 30 19:38:57.747166 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:57.746836    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa769f06-2fc2-40ce-bb1a-a198b2274afb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa769f06-2fc2-40ce-bb1a-a198b2274afb-istio-data\") pod \"aa769f06-2fc2-40ce-bb1a-a198b2274afb\" (UID: \"aa769f06-2fc2-40ce-bb1a-a198b2274afb\") "
Oct 30 19:38:57.747166 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:57.746964    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa769f06-2fc2-40ce-bb1a-a198b2274afb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa769f06-2fc2-40ce-bb1a-a198b2274afb-istio-podinfo\") pod \"aa769f06-2fc2-40ce-bb1a-a198b2274afb\" (UID: \"aa769f06-2fc2-40ce-bb1a-a198b2274afb\") "
Oct 30 19:38:57.747166 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:57.747009    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa769f06-2fc2-40ce-bb1a-a198b2274afb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa769f06-2fc2-40ce-bb1a-a198b2274afb-credential-socket\") pod \"aa769f06-2fc2-40ce-bb1a-a198b2274afb\" (UID: \"aa769f06-2fc2-40ce-bb1a-a198b2274afb\") "
Oct 30 19:38:57.747166 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:57.747096    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa769f06-2fc2-40ce-bb1a-a198b2274afb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa769f06-2fc2-40ce-bb1a-a198b2274afb-istiod-ca-cert\") pod \"aa769f06-2fc2-40ce-bb1a-a198b2274afb\" (UID: \"aa769f06-2fc2-40ce-bb1a-a198b2274afb\") "
Oct 30 19:38:57.747581 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:57.747193    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa769f06-2fc2-40ce-bb1a-a198b2274afb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa769f06-2fc2-40ce-bb1a-a198b2274afb-workload-certs\") pod \"aa769f06-2fc2-40ce-bb1a-a198b2274afb\" (UID: \"aa769f06-2fc2-40ce-bb1a-a198b2274afb\") "
Oct 30 19:38:57.747581 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:57.747258    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa769f06-2fc2-40ce-bb1a-a198b2274afb-kube-api-access-q5sd8\" (UniqueName: \"kubernetes.io/projected/aa769f06-2fc2-40ce-bb1a-a198b2274afb-kube-api-access-q5sd8\") pod \"aa769f06-2fc2-40ce-bb1a-a198b2274afb\" (UID: \"aa769f06-2fc2-40ce-bb1a-a198b2274afb\") "
Oct 30 19:38:57.747581 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:57.747308    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa769f06-2fc2-40ce-bb1a-a198b2274afb-istio-token\" (UniqueName: \"kubernetes.io/projected/aa769f06-2fc2-40ce-bb1a-a198b2274afb-istio-token\") pod \"aa769f06-2fc2-40ce-bb1a-a198b2274afb\" (UID: \"aa769f06-2fc2-40ce-bb1a-a198b2274afb\") "
Oct 30 19:38:57.747581 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:57.747420    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa769f06-2fc2-40ce-bb1a-a198b2274afb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa769f06-2fc2-40ce-bb1a-a198b2274afb-workload-socket\") pod \"aa769f06-2fc2-40ce-bb1a-a198b2274afb\" (UID: \"aa769f06-2fc2-40ce-bb1a-a198b2274afb\") "
Oct 30 19:38:57.748713 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:57.748360    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa769f06-2fc2-40ce-bb1a-a198b2274afb-istiod-ca-cert" pod "aa769f06-2fc2-40ce-bb1a-a198b2274afb" (UID: "aa769f06-2fc2-40ce-bb1a-a198b2274afb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:38:57.751424 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:57.751219    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa769f06-2fc2-40ce-bb1a-a198b2274afb-istio-envoy" pod "aa769f06-2fc2-40ce-bb1a-a198b2274afb" (UID: "aa769f06-2fc2-40ce-bb1a-a198b2274afb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:57.751897 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:57.751581    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa769f06-2fc2-40ce-bb1a-a198b2274afb-istio-data" pod "aa769f06-2fc2-40ce-bb1a-a198b2274afb" (UID: "aa769f06-2fc2-40ce-bb1a-a198b2274afb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:57.789050 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:57.788588    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa769f06-2fc2-40ce-bb1a-a198b2274afb-workload-certs" pod "aa769f06-2fc2-40ce-bb1a-a198b2274afb" (UID: "aa769f06-2fc2-40ce-bb1a-a198b2274afb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:57.789617 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:57.789286    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa769f06-2fc2-40ce-bb1a-a198b2274afb-workload-socket" pod "aa769f06-2fc2-40ce-bb1a-a198b2274afb" (UID: "aa769f06-2fc2-40ce-bb1a-a198b2274afb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:57.796613 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:57.792989    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa769f06-2fc2-40ce-bb1a-a198b2274afb-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:57.796613 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:57.793021    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa769f06-2fc2-40ce-bb1a-a198b2274afb-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:57.796613 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:57.793036    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa769f06-2fc2-40ce-bb1a-a198b2274afb-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:57.796613 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:57.793050    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa769f06-2fc2-40ce-bb1a-a198b2274afb-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:57.796613 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:57.793064    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa769f06-2fc2-40ce-bb1a-a198b2274afb-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:57.801256 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:57.798207    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa769f06-2fc2-40ce-bb1a-a198b2274afb-credential-socket" pod "aa769f06-2fc2-40ce-bb1a-a198b2274afb" (UID: "aa769f06-2fc2-40ce-bb1a-a198b2274afb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:38:57.801256 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:57.799268    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa769f06-2fc2-40ce-bb1a-a198b2274afb-kube-api-access-q5sd8" pod "aa769f06-2fc2-40ce-bb1a-a198b2274afb" (UID: "aa769f06-2fc2-40ce-bb1a-a198b2274afb"). InnerVolumeSpecName "kube-api-access-q5sd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:57.826602 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:57.826571    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-82012b-5987f89c59-xqtgs"]
Oct 30 19:38:57.838838 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:57.838811    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-82012b-5987f89c59-xqtgs"]
Oct 30 19:38:57.845245 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:57.845179    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ae425f-dzwmr"]
Oct 30 19:38:57.850825 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:57.849980    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa769f06-2fc2-40ce-bb1a-a198b2274afb-istio-podinfo" pod "aa769f06-2fc2-40ce-bb1a-a198b2274afb" (UID: "aa769f06-2fc2-40ce-bb1a-a198b2274afb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:38:57.855081 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:57.851147    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa769f06-2fc2-40ce-bb1a-a198b2274afb-istio-token" pod "aa769f06-2fc2-40ce-bb1a-a198b2274afb" (UID: "aa769f06-2fc2-40ce-bb1a-a198b2274afb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:57.931209 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:57.900919    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa769f06-2fc2-40ce-bb1a-a198b2274afb-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:57.931209 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:57.900959    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa769f06-2fc2-40ce-bb1a-a198b2274afb-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:57.931209 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:57.900986    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5sd8\" (UniqueName: \"kubernetes.io/projected/aa769f06-2fc2-40ce-bb1a-a198b2274afb-kube-api-access-q5sd8\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:57.931209 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:57.901002    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa769f06-2fc2-40ce-bb1a-a198b2274afb-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:38:57.931209 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:57.930384    5043 scope.go:122] "RemoveContainer" containerID="84556bc09334bd7c38da4f9ec3c9ba09f614085869b5048a421d05117f8210de"
Oct 30 19:38:57.931209 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:57.930877    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-ae425f-dzwmr"]
Oct 30 19:38:57.952189 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:57.952166    5043 scope.go:122] "RemoveContainer" containerID="3766a7f3ae3915d6bb7b92255e03f826433e7da7db7ec6e8be797782bd995437"
Oct 30 19:38:57.952628 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:38:57.952603    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3766a7f3ae3915d6bb7b92255e03f826433e7da7db7ec6e8be797782bd995437\": container with ID starting with 3766a7f3ae3915d6bb7b92255e03f826433e7da7db7ec6e8be797782bd995437 not found: ID does not exist" containerID="3766a7f3ae3915d6bb7b92255e03f826433e7da7db7ec6e8be797782bd995437"
Oct 30 19:38:57.952766 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:57.952636    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3766a7f3ae3915d6bb7b92255e03f826433e7da7db7ec6e8be797782bd995437"} err="failed to get container status \"3766a7f3ae3915d6bb7b92255e03f826433e7da7db7ec6e8be797782bd995437\": rpc error: code = NotFound desc = could not find container \"3766a7f3ae3915d6bb7b92255e03f826433e7da7db7ec6e8be797782bd995437\": container with ID starting with 3766a7f3ae3915d6bb7b92255e03f826433e7da7db7ec6e8be797782bd995437 not found: ID does not exist"
Oct 30 19:38:57.952766 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:57.952656    5043 scope.go:122] "RemoveContainer" containerID="84556bc09334bd7c38da4f9ec3c9ba09f614085869b5048a421d05117f8210de"
Oct 30 19:38:57.953014 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:38:57.952993    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84556bc09334bd7c38da4f9ec3c9ba09f614085869b5048a421d05117f8210de\": container with ID starting with 84556bc09334bd7c38da4f9ec3c9ba09f614085869b5048a421d05117f8210de not found: ID does not exist" containerID="84556bc09334bd7c38da4f9ec3c9ba09f614085869b5048a421d05117f8210de"
Oct 30 19:38:57.953149 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:57.953133    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84556bc09334bd7c38da4f9ec3c9ba09f614085869b5048a421d05117f8210de"} err="failed to get container status \"84556bc09334bd7c38da4f9ec3c9ba09f614085869b5048a421d05117f8210de\": rpc error: code = NotFound desc = could not find container \"84556bc09334bd7c38da4f9ec3c9ba09f614085869b5048a421d05117f8210de\": container with ID starting with 84556bc09334bd7c38da4f9ec3c9ba09f614085869b5048a421d05117f8210de not found: ID does not exist"
Oct 30 19:38:58.195819 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:58.195469    5043 patch_prober.go:29] interesting pod/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:38:58.195819 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:58.195542    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5" podUID="aa769f06-2fc2-40ce-bb1a-a198b2274afb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:38:58.648225 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:58.648191    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5"
Oct 30 19:38:58.686761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:58.685773    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="346d6503-7d76-47c4-8528-18d1f0669680" path="/var/lib/kubelet/pods/346d6503-7d76-47c4-8528-18d1f0669680/volumes"
Oct 30 19:38:58.686761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:58.686654    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d41c7166-ac32-4778-8af3-3d32819427e3" path="/var/lib/kubelet/pods/d41c7166-ac32-4778-8af3-3d32819427e3/volumes"
Oct 30 19:38:58.826596 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:58.826561    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5"]
Oct 30 19:38:58.831411 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:58.831381    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-6a6348-openshift-default-7bd9c5bc47-cmtk5"]
Oct 30 19:38:59.228103 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:59.228067    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-744b57-77cc66bfc9-2fcz9"]
Oct 30 19:38:59.229596 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:59.229565    5043 cpu_manager.go:414] "containerMap: removing container" podUID="aa769f06-2fc2-40ce-bb1a-a198b2274afb" containerName="istio-proxy"
Oct 30 19:38:59.233389 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:59.233363    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa769f06-2fc2-40ce-bb1a-a198b2274afb" containerName="istio-proxy"
Oct 30 19:38:59.233637 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:59.233626    5043 cpu_manager.go:414] "containerMap: removing container" podUID="d41c7166-ac32-4778-8af3-3d32819427e3" containerName="payload-pre-processing"
Oct 30 19:38:59.233753 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:59.233721    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d41c7166-ac32-4778-8af3-3d32819427e3" containerName="payload-pre-processing"
Oct 30 19:38:59.233945 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:59.233929    5043 cpu_manager.go:414] "containerMap: removing container" podUID="7753bd1c-f916-40db-a923-eb46e92a548e" containerName="payload-pre-processing"
Oct 30 19:38:59.234094 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:59.234062    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7753bd1c-f916-40db-a923-eb46e92a548e" containerName="payload-pre-processing"
Oct 30 19:38:59.234226 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:59.234215    5043 cpu_manager.go:414] "containerMap: removing container" podUID="469e981b-1193-4d89-b98a-bc6f794cb532" containerName="payload-processing"
Oct 30 19:38:59.234341 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:59.234330    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="469e981b-1193-4d89-b98a-bc6f794cb532" containerName="payload-processing"
Oct 30 19:38:59.234695 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:59.234681    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="7753bd1c-f916-40db-a923-eb46e92a548e" containerName="payload-pre-processing"
Oct 30 19:38:59.234858 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:59.234846    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="469e981b-1193-4d89-b98a-bc6f794cb532" containerName="payload-processing"
Oct 30 19:38:59.236094 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:59.236062    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa769f06-2fc2-40ce-bb1a-a198b2274afb" containerName="istio-proxy"
Oct 30 19:38:59.236277 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:59.236264    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="d41c7166-ac32-4778-8af3-3d32819427e3" containerName="payload-pre-processing"
Oct 30 19:38:59.236419 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:59.236407    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa769f06-2fc2-40ce-bb1a-a198b2274afb" containerName="istio-proxy"
Oct 30 19:38:59.272762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:59.267744    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-744b57-77cc66bfc9-2fcz9"]
Oct 30 19:38:59.280766 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:59.275803    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-744b57-77cc66bfc9-2fcz9"
Oct 30 19:38:59.307994 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:59.307962    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-744b57\""
Oct 30 19:38:59.309209 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:59.309186    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-744b57-dockercfg-kkcvl\""
Oct 30 19:38:59.372053 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:59.371950    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ef41111-49f3-4fd9-9587-2a1934f50797-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-744b57-77cc66bfc9-2fcz9\" (UID: \"4ef41111-49f3-4fd9-9587-2a1934f50797\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-744b57-77cc66bfc9-2fcz9"
Oct 30 19:38:59.372053 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:59.372035    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwqtn\" (UniqueName: \"kubernetes.io/projected/4ef41111-49f3-4fd9-9587-2a1934f50797-kube-api-access-xwqtn\") pod \"payload-pre-processing-e2e-inf-a-744b57-77cc66bfc9-2fcz9\" (UID: \"4ef41111-49f3-4fd9-9587-2a1934f50797\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-744b57-77cc66bfc9-2fcz9"
Oct 30 19:38:59.465177 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:59.465150    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-744b57-597d6c8986-4v2zh"]
Oct 30 19:38:59.493810 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:59.474945    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ef41111-49f3-4fd9-9587-2a1934f50797-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-744b57-77cc66bfc9-2fcz9\" (UID: \"4ef41111-49f3-4fd9-9587-2a1934f50797\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-744b57-77cc66bfc9-2fcz9"
Oct 30 19:38:59.493810 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:59.475027    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwqtn\" (UniqueName: \"kubernetes.io/projected/4ef41111-49f3-4fd9-9587-2a1934f50797-kube-api-access-xwqtn\") pod \"payload-pre-processing-e2e-inf-a-744b57-77cc66bfc9-2fcz9\" (UID: \"4ef41111-49f3-4fd9-9587-2a1934f50797\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-744b57-77cc66bfc9-2fcz9"
Oct 30 19:38:59.493810 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:59.477473    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ef41111-49f3-4fd9-9587-2a1934f50797-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-744b57-77cc66bfc9-2fcz9\" (UID: \"4ef41111-49f3-4fd9-9587-2a1934f50797\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-744b57-77cc66bfc9-2fcz9"
Oct 30 19:38:59.584987 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:59.511085    5043 cpu_manager.go:414] "containerMap: removing container" podUID="aa769f06-2fc2-40ce-bb1a-a198b2274afb" containerName="istio-proxy"
Oct 30 19:38:59.584987 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:59.511121    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa769f06-2fc2-40ce-bb1a-a198b2274afb" containerName="istio-proxy"
Oct 30 19:38:59.584987 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:59.529629    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-744b57-597d6c8986-4v2zh"
Oct 30 19:38:59.584987 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:59.580433    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwqtn\" (UniqueName: \"kubernetes.io/projected/4ef41111-49f3-4fd9-9587-2a1934f50797-kube-api-access-xwqtn\") pod \"payload-pre-processing-e2e-inf-a-744b57-77cc66bfc9-2fcz9\" (UID: \"4ef41111-49f3-4fd9-9587-2a1934f50797\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-744b57-77cc66bfc9-2fcz9"
Oct 30 19:38:59.739883 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:59.698488    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42404324-3c5e-4842-abad-eeb53637016e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-744b57-597d6c8986-4v2zh\" (UID: \"42404324-3c5e-4842-abad-eeb53637016e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-744b57-597d6c8986-4v2zh"
Oct 30 19:38:59.761810 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:59.755955    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9k9m\" (UniqueName: \"kubernetes.io/projected/42404324-3c5e-4842-abad-eeb53637016e-kube-api-access-k9k9m\") pod \"payload-processing-e2e-inf-a-744b57-597d6c8986-4v2zh\" (UID: \"42404324-3c5e-4842-abad-eeb53637016e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-744b57-597d6c8986-4v2zh"
Oct 30 19:38:59.761810 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:59.739807    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-744b57-597d6c8986-4v2zh"]
Oct 30 19:38:59.842870 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:59.842843    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-744b57-77cc66bfc9-2fcz9"
Oct 30 19:38:59.876774 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:59.858241    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42404324-3c5e-4842-abad-eeb53637016e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-744b57-597d6c8986-4v2zh\" (UID: \"42404324-3c5e-4842-abad-eeb53637016e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-744b57-597d6c8986-4v2zh"
Oct 30 19:38:59.876774 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:59.858307    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9k9m\" (UniqueName: \"kubernetes.io/projected/42404324-3c5e-4842-abad-eeb53637016e-kube-api-access-k9k9m\") pod \"payload-processing-e2e-inf-a-744b57-597d6c8986-4v2zh\" (UID: \"42404324-3c5e-4842-abad-eeb53637016e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-744b57-597d6c8986-4v2zh"
Oct 30 19:38:59.876774 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:59.859658    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42404324-3c5e-4842-abad-eeb53637016e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-744b57-597d6c8986-4v2zh\" (UID: \"42404324-3c5e-4842-abad-eeb53637016e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-744b57-597d6c8986-4v2zh"
Oct 30 19:38:59.934783 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:59.933596    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9k9m\" (UniqueName: \"kubernetes.io/projected/42404324-3c5e-4842-abad-eeb53637016e-kube-api-access-k9k9m\") pod \"payload-processing-e2e-inf-a-744b57-597d6c8986-4v2zh\" (UID: \"42404324-3c5e-4842-abad-eeb53637016e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-744b57-597d6c8986-4v2zh"
Oct 30 19:38:59.947111 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:38:59.947085    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-744b57-597d6c8986-4v2zh"
Oct 30 19:39:00.150285 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:00.150206    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-22af66-5bf7f46cd7-ffx8p"
Oct 30 19:39:00.841258 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:00.838469    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa769f06-2fc2-40ce-bb1a-a198b2274afb" path="/var/lib/kubelet/pods/aa769f06-2fc2-40ce-bb1a-a198b2274afb/volumes"
Oct 30 19:39:01.030766 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:01.028463    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-22af66-7b5b988746-mm9zl"
Oct 30 19:39:01.038971 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:01.038945    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-744b57-597d6c8986-4v2zh"]
Oct 30 19:39:01.042558 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:01.042524    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-744b57-77cc66bfc9-2fcz9"]
Oct 30 19:39:01.042675 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:39:01.042647    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42404324_3c5e_4842_abad_eeb53637016e.slice/crio-8ebc712d937210811483acb7644a8c493fa2d787e7e2b41307322c7a3c1e9c8e WatchSource:0}: Error finding container 8ebc712d937210811483acb7644a8c493fa2d787e7e2b41307322c7a3c1e9c8e: Status 404 returned error can't find the container with id 8ebc712d937210811483acb7644a8c493fa2d787e7e2b41307322c7a3c1e9c8e
Oct 30 19:39:01.881077 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:01.881038    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-744b57-77cc66bfc9-2fcz9" event={"ID":"4ef41111-49f3-4fd9-9587-2a1934f50797","Type":"ContainerStarted","Data":"af0969d1dcc550e1c71ac468872d2556fec54bb81e7596638a935a164561936c"}
Oct 30 19:39:01.881572 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:01.881106    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-744b57-77cc66bfc9-2fcz9" event={"ID":"4ef41111-49f3-4fd9-9587-2a1934f50797","Type":"ContainerStarted","Data":"f9d9e69c15cc1f0edc4a4f95d4b43647b87e9f559d96736226c9f3224b8d69bc"}
Oct 30 19:39:01.881881 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:01.881858    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-744b57-77cc66bfc9-2fcz9"
Oct 30 19:39:01.883258 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:01.883209    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-744b57-597d6c8986-4v2zh" event={"ID":"42404324-3c5e-4842-abad-eeb53637016e","Type":"ContainerStarted","Data":"763c854283902c6ac04af0d967742004a31067845c7da719362e06ba090a1132"}
Oct 30 19:39:01.883258 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:01.883248    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-744b57-597d6c8986-4v2zh" event={"ID":"42404324-3c5e-4842-abad-eeb53637016e","Type":"ContainerStarted","Data":"8ebc712d937210811483acb7644a8c493fa2d787e7e2b41307322c7a3c1e9c8e"}
Oct 30 19:39:01.883678 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:01.883659    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-744b57-597d6c8986-4v2zh"
Oct 30 19:39:01.910312 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:01.910241    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-744b57-77cc66bfc9-2fcz9" podStartSLOduration=2.910222729 podStartE2EDuration="2.910222729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:01.90434945 +0000 UTC m=+1361.415409414" watchObservedRunningTime="2026-10-30 19:39:01.910222729 +0000 UTC m=+1361.421282696"
Oct 30 19:39:01.928068 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:01.928015    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-744b57-597d6c8986-4v2zh" podStartSLOduration=2.928002046 podStartE2EDuration="2.928002046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:01.924118056 +0000 UTC m=+1361.435178022" watchObservedRunningTime="2026-10-30 19:39:01.928002046 +0000 UTC m=+1361.439062005"
Oct 30 19:39:06.476549 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:06.463779    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf"]
Oct 30 19:39:06.476549 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:06.464028    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf" podUID="40748d3a-8823-40ee-9425-f5e36a8889d6" containerName="istio-proxy" containerID="cri-o://cf42adb3acbf7ae83c1c9c86b507995dbef94847b2237dafaa692a1f4752e02a" gracePeriod=30
Oct 30 19:39:07.412879 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:07.317203    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ae63bf-5xlsr"]
Oct 30 19:39:07.412879 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:07.328526    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ae63bf-5xlsr"
Oct 30 19:39:07.412879 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:07.330984    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ae63bf-5xlsr"]
Oct 30 19:39:07.412879 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:07.332019    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:39:07.412879 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:07.332859    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:39:07.412879 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:07.398097    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:39:07.412879 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:07.398140    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:39:07.412879 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:07.404856    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhjfl\" (UniqueName: \"kubernetes.io/projected/39cf0453-c5af-4fba-8a9c-5d3ace5dc747-kube-api-access-lhjfl\") pod \"maas-api-revoke-keys-e2e-worker-w0-ae63bf-5xlsr\" (UID: \"39cf0453-c5af-4fba-8a9c-5d3ace5dc747\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ae63bf-5xlsr"
Oct 30 19:39:07.412879 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:07.404921    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39cf0453-c5af-4fba-8a9c-5d3ace5dc747-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ae63bf-5xlsr\" (UID: \"39cf0453-c5af-4fba-8a9c-5d3ace5dc747\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ae63bf-5xlsr"
Oct 30 19:39:07.412879 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:07.405091    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39cf0453-c5af-4fba-8a9c-5d3ace5dc747-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ae63bf-5xlsr\" (UID: \"39cf0453-c5af-4fba-8a9c-5d3ace5dc747\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ae63bf-5xlsr"
Oct 30 19:39:07.506496 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:07.506452    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39cf0453-c5af-4fba-8a9c-5d3ace5dc747-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ae63bf-5xlsr\" (UID: \"39cf0453-c5af-4fba-8a9c-5d3ace5dc747\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ae63bf-5xlsr"
Oct 30 19:39:07.506925 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:07.506585    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39cf0453-c5af-4fba-8a9c-5d3ace5dc747-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ae63bf-5xlsr\" (UID: \"39cf0453-c5af-4fba-8a9c-5d3ace5dc747\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ae63bf-5xlsr"
Oct 30 19:39:07.506925 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:07.506869    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhjfl\" (UniqueName: \"kubernetes.io/projected/39cf0453-c5af-4fba-8a9c-5d3ace5dc747-kube-api-access-lhjfl\") pod \"maas-api-revoke-keys-e2e-worker-w0-ae63bf-5xlsr\" (UID: \"39cf0453-c5af-4fba-8a9c-5d3ace5dc747\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ae63bf-5xlsr"
Oct 30 19:39:07.507214 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:07.507187    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39cf0453-c5af-4fba-8a9c-5d3ace5dc747-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ae63bf-5xlsr\" (UID: \"39cf0453-c5af-4fba-8a9c-5d3ace5dc747\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ae63bf-5xlsr"
Oct 30 19:39:07.509229 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:07.509202    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39cf0453-c5af-4fba-8a9c-5d3ace5dc747-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ae63bf-5xlsr\" (UID: \"39cf0453-c5af-4fba-8a9c-5d3ace5dc747\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ae63bf-5xlsr"
Oct 30 19:39:07.516625 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:07.516595    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhjfl\" (UniqueName: \"kubernetes.io/projected/39cf0453-c5af-4fba-8a9c-5d3ace5dc747-kube-api-access-lhjfl\") pod \"maas-api-revoke-keys-e2e-worker-w0-ae63bf-5xlsr\" (UID: \"39cf0453-c5af-4fba-8a9c-5d3ace5dc747\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ae63bf-5xlsr"
Oct 30 19:39:07.661642 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:07.661247    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ae63bf-5xlsr"
Oct 30 19:39:07.954940 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:07.954869    5043 generic.go:356] "Generic (PLEG): container finished" podID="2db29833-fd7c-4dfd-88ba-6ae264d9f8b5" containerID="bf77081744284b6717031a6400821be62368290fa84e5f36bf4da82dcffa73b4" exitCode=0
Oct 30 19:39:07.955095 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:07.954940    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd" event={"ID":"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5","Type":"ContainerDied","Data":"bf77081744284b6717031a6400821be62368290fa84e5f36bf4da82dcffa73b4"}
Oct 30 19:39:07.955440 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:07.955422    5043 scope.go:122] "RemoveContainer" containerID="bf77081744284b6717031a6400821be62368290fa84e5f36bf4da82dcffa73b4"
Oct 30 19:39:07.962250 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:07.957334    5043 generic.go:356] "Generic (PLEG): container finished" podID="a74dccf1-3de7-4f70-a549-281361992615" containerID="27ac769c108e2a9c3268d15f324d53c35afa80c74599451c9d15b3ea126d1e9e" exitCode=0
Oct 30 19:39:07.962250 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:07.957365    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp" event={"ID":"a74dccf1-3de7-4f70-a549-281361992615","Type":"ContainerDied","Data":"27ac769c108e2a9c3268d15f324d53c35afa80c74599451c9d15b3ea126d1e9e"}
Oct 30 19:39:07.962250 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:07.957461    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:39:07.962250 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:07.957526    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:39:07.962250 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:07.957556    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:39:07.962250 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:07.957836    5043 scope.go:122] "RemoveContainer" containerID="27ac769c108e2a9c3268d15f324d53c35afa80c74599451c9d15b3ea126d1e9e"
Oct 30 19:39:07.962250 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:07.959572    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:39:07.962250 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:07.959628    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:39:07.962250 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:07.959658    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:39:07.980843 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:07.980817    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ae63bf-5xlsr"]
Oct 30 19:39:08.004893 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:39:08.004867    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39cf0453_c5af_4fba_8a9c_5d3ace5dc747.slice/crio-f2cea356bf085be2322c0a243b36582ca58da5c63fee597c67298ce88670dbae WatchSource:0}: Error finding container f2cea356bf085be2322c0a243b36582ca58da5c63fee597c67298ce88670dbae: Status 404 returned error can't find the container with id f2cea356bf085be2322c0a243b36582ca58da5c63fee597c67298ce88670dbae
Oct 30 19:39:08.404706 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:08.404659    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd"
Oct 30 19:39:08.732904 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:08.732869    5043 patch_prober.go:29] interesting pod/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:39:08.733347 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:08.732935    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf" podUID="40748d3a-8823-40ee-9425-f5e36a8889d6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:39:09.017819 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:09.017472    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ae63bf-5xlsr" event={"ID":"39cf0453-c5af-4fba-8a9c-5d3ace5dc747","Type":"ContainerStarted","Data":"c635515c328b8d3f93e621ab8e761c5c2f5316c99e9fc4184214454186943912"}
Oct 30 19:39:09.017819 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:09.017548    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ae63bf-5xlsr" event={"ID":"39cf0453-c5af-4fba-8a9c-5d3ace5dc747","Type":"ContainerStarted","Data":"f2cea356bf085be2322c0a243b36582ca58da5c63fee597c67298ce88670dbae"}
Oct 30 19:39:09.025153 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:09.025121    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp" event={"ID":"a74dccf1-3de7-4f70-a549-281361992615","Type":"ContainerStarted","Data":"a9ba0d5574dfb6107ce27928d476f6e1716cc4ae0765f3742efbf5c3ce01a001"}
Oct 30 19:39:09.028692 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:09.028661    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd" event={"ID":"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5","Type":"ContainerStarted","Data":"93d64382b6282c4799809e0405a91470ae3f723232661dcb860d85da7b7b62e4"}
Oct 30 19:39:09.048023 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:09.047957    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ae63bf-5xlsr" podStartSLOduration=2.047939272 podStartE2EDuration="2.047939272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:09.038789107 +0000 UTC m=+1368.549849144" watchObservedRunningTime="2026-10-30 19:39:09.047939272 +0000 UTC m=+1368.558999240"
Oct 30 19:39:09.329017 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:09.328837    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp"
Oct 30 19:39:09.329929 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:09.329899    5043 patch_prober.go:29] interesting pod/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Oct 30 19:39:09.330035 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:09.329961    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp" podUID="a74dccf1-3de7-4f70-a549-281361992615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Oct 30 19:39:09.532521 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:09.532128    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd"
Oct 30 19:39:09.537783 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:09.537558    5043 patch_prober.go:29] interesting pod/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Oct 30 19:39:09.537783 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:09.537658    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd" podUID="2db29833-fd7c-4dfd-88ba-6ae264d9f8b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Oct 30 19:39:10.042136 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:10.042101    5043 generic.go:356] "Generic (PLEG): container finished" podID="39cf0453-c5af-4fba-8a9c-5d3ace5dc747" containerID="c635515c328b8d3f93e621ab8e761c5c2f5316c99e9fc4184214454186943912" exitCode=0
Oct 30 19:39:10.042801 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:10.042772    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ae63bf-5xlsr" event={"ID":"39cf0453-c5af-4fba-8a9c-5d3ace5dc747","Type":"ContainerDied","Data":"c635515c328b8d3f93e621ab8e761c5c2f5316c99e9fc4184214454186943912"}
Oct 30 19:39:10.329031 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:10.328949    5043 patch_prober.go:29] interesting pod/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Oct 30 19:39:10.329031 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:10.328998    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp" podUID="a74dccf1-3de7-4f70-a549-281361992615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Oct 30 19:39:10.530405 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:10.530357    5043 patch_prober.go:29] interesting pod/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Oct 30 19:39:10.530589 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:10.530413    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd" podUID="2db29833-fd7c-4dfd-88ba-6ae264d9f8b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Oct 30 19:39:11.358013 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:11.357480    5043 patch_prober.go:29] interesting pod/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Oct 30 19:39:11.381761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:11.378980    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp" podUID="a74dccf1-3de7-4f70-a549-281361992615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Oct 30 19:39:11.649138 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:11.580871    5043 patch_prober.go:29] interesting pod/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Oct 30 19:39:11.649138 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:11.580940    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd" podUID="2db29833-fd7c-4dfd-88ba-6ae264d9f8b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Oct 30 19:39:11.739601 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:11.739565    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ae63bf-5xlsr"
Oct 30 19:39:11.813768 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:11.813350    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39cf0453-c5af-4fba-8a9c-5d3ace5dc747-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39cf0453-c5af-4fba-8a9c-5d3ace5dc747-maas-api-xks-ca\") pod \"39cf0453-c5af-4fba-8a9c-5d3ace5dc747\" (UID: \"39cf0453-c5af-4fba-8a9c-5d3ace5dc747\") "
Oct 30 19:39:11.813768 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:11.813546    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39cf0453-c5af-4fba-8a9c-5d3ace5dc747-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39cf0453-c5af-4fba-8a9c-5d3ace5dc747-maas-api-openshift-ca\") pod \"39cf0453-c5af-4fba-8a9c-5d3ace5dc747\" (UID: \"39cf0453-c5af-4fba-8a9c-5d3ace5dc747\") "
Oct 30 19:39:11.813768 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:11.813686    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39cf0453-c5af-4fba-8a9c-5d3ace5dc747-kube-api-access-lhjfl\" (UniqueName: \"kubernetes.io/projected/39cf0453-c5af-4fba-8a9c-5d3ace5dc747-kube-api-access-lhjfl\") pod \"39cf0453-c5af-4fba-8a9c-5d3ace5dc747\" (UID: \"39cf0453-c5af-4fba-8a9c-5d3ace5dc747\") "
Oct 30 19:39:11.826791 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:11.815434    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39cf0453-c5af-4fba-8a9c-5d3ace5dc747-maas-api-openshift-ca" pod "39cf0453-c5af-4fba-8a9c-5d3ace5dc747" (UID: "39cf0453-c5af-4fba-8a9c-5d3ace5dc747"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:11.826791 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:11.816619    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39cf0453-c5af-4fba-8a9c-5d3ace5dc747-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:11.855033 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:11.854975    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39cf0453-c5af-4fba-8a9c-5d3ace5dc747-kube-api-access-lhjfl" pod "39cf0453-c5af-4fba-8a9c-5d3ace5dc747" (UID: "39cf0453-c5af-4fba-8a9c-5d3ace5dc747"). InnerVolumeSpecName "kube-api-access-lhjfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:11.937866 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:11.935946    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39cf0453-c5af-4fba-8a9c-5d3ace5dc747-maas-api-xks-ca" pod "39cf0453-c5af-4fba-8a9c-5d3ace5dc747" (UID: "39cf0453-c5af-4fba-8a9c-5d3ace5dc747"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:39:11.937866 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:11.936614    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39cf0453-c5af-4fba-8a9c-5d3ace5dc747-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:11.937866 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:11.936644    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhjfl\" (UniqueName: \"kubernetes.io/projected/39cf0453-c5af-4fba-8a9c-5d3ace5dc747-kube-api-access-lhjfl\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:11.948777 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:11.948709    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"]
Oct 30 19:39:11.949655 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:11.949621    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"]
Oct 30 19:39:12.024773 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:11.957266    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"]
Oct 30 19:39:12.344708 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:12.344472    5043 patch_prober.go:29] interesting pod/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Oct 30 19:39:12.349459 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:12.349408    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp" podUID="a74dccf1-3de7-4f70-a549-281361992615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Oct 30 19:39:12.427826 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:12.427450    5043 generic.go:356] "Generic (PLEG): container finished" podID="40748d3a-8823-40ee-9425-f5e36a8889d6" containerID="cf42adb3acbf7ae83c1c9c86b507995dbef94847b2237dafaa692a1f4752e02a" exitCode=0
Oct 30 19:39:12.427826 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:12.427571    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf" event={"ID":"40748d3a-8823-40ee-9425-f5e36a8889d6","Type":"ContainerDied","Data":"cf42adb3acbf7ae83c1c9c86b507995dbef94847b2237dafaa692a1f4752e02a"}
Oct 30 19:39:12.427826 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:12.427676    5043 scope.go:122] "RemoveContainer" containerID="90c4e9cd64ebdd562a5dc406f8343f7b669002370aa53934d44370f6fcf30262"
Oct 30 19:39:12.437243 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:12.437209    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ae63bf-5xlsr" event={"ID":"39cf0453-c5af-4fba-8a9c-5d3ace5dc747","Type":"ContainerDied","Data":"f2cea356bf085be2322c0a243b36582ca58da5c63fee597c67298ce88670dbae"}
Oct 30 19:39:12.437360 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:12.437269    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2cea356bf085be2322c0a243b36582ca58da5c63fee597c67298ce88670dbae"
Oct 30 19:39:12.437360 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:12.437352    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ae63bf-5xlsr"
Oct 30 19:39:12.446896 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:12.446871    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf"
Oct 30 19:39:12.524572 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:12.523427    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40748d3a-8823-40ee-9425-f5e36a8889d6-istio-token\" (UniqueName: \"kubernetes.io/projected/40748d3a-8823-40ee-9425-f5e36a8889d6-istio-token\") pod \"40748d3a-8823-40ee-9425-f5e36a8889d6\" (UID: \"40748d3a-8823-40ee-9425-f5e36a8889d6\") "
Oct 30 19:39:12.524572 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:12.523462    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40748d3a-8823-40ee-9425-f5e36a8889d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40748d3a-8823-40ee-9425-f5e36a8889d6-workload-socket\") pod \"40748d3a-8823-40ee-9425-f5e36a8889d6\" (UID: \"40748d3a-8823-40ee-9425-f5e36a8889d6\") "
Oct 30 19:39:12.524572 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:12.523513    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40748d3a-8823-40ee-9425-f5e36a8889d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40748d3a-8823-40ee-9425-f5e36a8889d6-istiod-ca-cert\") pod \"40748d3a-8823-40ee-9425-f5e36a8889d6\" (UID: \"40748d3a-8823-40ee-9425-f5e36a8889d6\") "
Oct 30 19:39:12.524572 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:12.523620    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40748d3a-8823-40ee-9425-f5e36a8889d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40748d3a-8823-40ee-9425-f5e36a8889d6-workload-certs\") pod \"40748d3a-8823-40ee-9425-f5e36a8889d6\" (UID: \"40748d3a-8823-40ee-9425-f5e36a8889d6\") "
Oct 30 19:39:12.524572 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:12.523670    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40748d3a-8823-40ee-9425-f5e36a8889d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40748d3a-8823-40ee-9425-f5e36a8889d6-istio-data\") pod \"40748d3a-8823-40ee-9425-f5e36a8889d6\" (UID: \"40748d3a-8823-40ee-9425-f5e36a8889d6\") "
Oct 30 19:39:12.524572 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:12.523696    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40748d3a-8823-40ee-9425-f5e36a8889d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40748d3a-8823-40ee-9425-f5e36a8889d6-istio-podinfo\") pod \"40748d3a-8823-40ee-9425-f5e36a8889d6\" (UID: \"40748d3a-8823-40ee-9425-f5e36a8889d6\") "
Oct 30 19:39:12.524572 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:12.523821    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40748d3a-8823-40ee-9425-f5e36a8889d6-kube-api-access-9dxzs\" (UniqueName: \"kubernetes.io/projected/40748d3a-8823-40ee-9425-f5e36a8889d6-kube-api-access-9dxzs\") pod \"40748d3a-8823-40ee-9425-f5e36a8889d6\" (UID: \"40748d3a-8823-40ee-9425-f5e36a8889d6\") "
Oct 30 19:39:12.524572 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:12.523885    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40748d3a-8823-40ee-9425-f5e36a8889d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40748d3a-8823-40ee-9425-f5e36a8889d6-istio-envoy\") pod \"40748d3a-8823-40ee-9425-f5e36a8889d6\" (UID: \"40748d3a-8823-40ee-9425-f5e36a8889d6\") "
Oct 30 19:39:12.524572 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:12.523936    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40748d3a-8823-40ee-9425-f5e36a8889d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40748d3a-8823-40ee-9425-f5e36a8889d6-credential-socket\") pod \"40748d3a-8823-40ee-9425-f5e36a8889d6\" (UID: \"40748d3a-8823-40ee-9425-f5e36a8889d6\") "
Oct 30 19:39:12.525025 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:12.524583    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40748d3a-8823-40ee-9425-f5e36a8889d6-credential-socket" pod "40748d3a-8823-40ee-9425-f5e36a8889d6" (UID: "40748d3a-8823-40ee-9425-f5e36a8889d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:12.525369 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:12.525348    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40748d3a-8823-40ee-9425-f5e36a8889d6-workload-socket" pod "40748d3a-8823-40ee-9425-f5e36a8889d6" (UID: "40748d3a-8823-40ee-9425-f5e36a8889d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:12.525581 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:12.525566    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40748d3a-8823-40ee-9425-f5e36a8889d6-istiod-ca-cert" pod "40748d3a-8823-40ee-9425-f5e36a8889d6" (UID: "40748d3a-8823-40ee-9425-f5e36a8889d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:12.525761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:12.525723    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40748d3a-8823-40ee-9425-f5e36a8889d6-workload-certs" pod "40748d3a-8823-40ee-9425-f5e36a8889d6" (UID: "40748d3a-8823-40ee-9425-f5e36a8889d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:12.525889 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:12.525875    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40748d3a-8823-40ee-9425-f5e36a8889d6-istio-data" pod "40748d3a-8823-40ee-9425-f5e36a8889d6" (UID: "40748d3a-8823-40ee-9425-f5e36a8889d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:12.527780 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:12.527721    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40748d3a-8823-40ee-9425-f5e36a8889d6-istio-token" pod "40748d3a-8823-40ee-9425-f5e36a8889d6" (UID: "40748d3a-8823-40ee-9425-f5e36a8889d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:12.528935 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:12.528664    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40748d3a-8823-40ee-9425-f5e36a8889d6-istio-podinfo" pod "40748d3a-8823-40ee-9425-f5e36a8889d6" (UID: "40748d3a-8823-40ee-9425-f5e36a8889d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:39:12.528935 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:12.528676    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40748d3a-8823-40ee-9425-f5e36a8889d6-kube-api-access-9dxzs" pod "40748d3a-8823-40ee-9425-f5e36a8889d6" (UID: "40748d3a-8823-40ee-9425-f5e36a8889d6"). InnerVolumeSpecName "kube-api-access-9dxzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:12.529847 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:12.529316    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40748d3a-8823-40ee-9425-f5e36a8889d6-istio-envoy" pod "40748d3a-8823-40ee-9425-f5e36a8889d6" (UID: "40748d3a-8823-40ee-9425-f5e36a8889d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:12.530982 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:12.530959    5043 patch_prober.go:29] interesting pod/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Oct 30 19:39:12.531085 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:12.531009    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd" podUID="2db29833-fd7c-4dfd-88ba-6ae264d9f8b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Oct 30 19:39:12.625995 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:12.625968    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dxzs\" (UniqueName: \"kubernetes.io/projected/40748d3a-8823-40ee-9425-f5e36a8889d6-kube-api-access-9dxzs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:12.625995 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:12.625993    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40748d3a-8823-40ee-9425-f5e36a8889d6-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:12.626182 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:12.626005    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40748d3a-8823-40ee-9425-f5e36a8889d6-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:12.626182 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:12.626014    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40748d3a-8823-40ee-9425-f5e36a8889d6-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:12.626182 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:12.626022    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40748d3a-8823-40ee-9425-f5e36a8889d6-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:12.626182 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:12.626030    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40748d3a-8823-40ee-9425-f5e36a8889d6-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:12.626182 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:12.626038    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40748d3a-8823-40ee-9425-f5e36a8889d6-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:12.626182 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:12.626046    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40748d3a-8823-40ee-9425-f5e36a8889d6-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:12.626182 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:12.626054    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40748d3a-8823-40ee-9425-f5e36a8889d6-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:13.330195 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:13.330155    5043 patch_prober.go:29] interesting pod/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Oct 30 19:39:13.330507 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:13.330476    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp" podUID="a74dccf1-3de7-4f70-a549-281361992615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Oct 30 19:39:13.455044 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:13.455013    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf"
Oct 30 19:39:13.455486 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:13.455055    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf" event={"ID":"40748d3a-8823-40ee-9425-f5e36a8889d6","Type":"ContainerDied","Data":"d7d229d468c17396abf169c27db7a18d9eea44ddf701899963d25fc37385d84c"}
Oct 30 19:39:13.455486 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:13.455122    5043 scope.go:122] "RemoveContainer" containerID="cf42adb3acbf7ae83c1c9c86b507995dbef94847b2237dafaa692a1f4752e02a"
Oct 30 19:39:13.529344 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:13.529308    5043 patch_prober.go:29] interesting pod/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Oct 30 19:39:13.529495 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:13.529374    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd" podUID="2db29833-fd7c-4dfd-88ba-6ae264d9f8b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Oct 30 19:39:13.536350 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:13.536318    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf"]
Oct 30 19:39:13.541700 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:13.541671    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-ae425f-openshift-default-76874f9f6c-vszwf"]
Oct 30 19:39:13.831709 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:13.831675    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp"
Oct 30 19:39:13.923492 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:13.923464    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-744b57-77cc66bfc9-2fcz9"
Oct 30 19:39:13.924163 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:13.924145    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-744b57-597d6c8986-4v2zh"
Oct 30 19:39:14.329013 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:14.328971    5043 patch_prober.go:29] interesting pod/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Oct 30 19:39:14.329196 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:14.329028    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp" podUID="a74dccf1-3de7-4f70-a549-281361992615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Oct 30 19:39:14.529019 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:14.528982    5043 patch_prober.go:29] interesting pod/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Oct 30 19:39:14.529561 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:14.529048    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd" podUID="2db29833-fd7c-4dfd-88ba-6ae264d9f8b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Oct 30 19:39:14.659015 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:14.658933    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40748d3a-8823-40ee-9425-f5e36a8889d6" path="/var/lib/kubelet/pods/40748d3a-8823-40ee-9425-f5e36a8889d6/volumes"
Oct 30 19:39:15.333060 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:15.333034    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp"
Oct 30 19:39:15.334118 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:15.334092    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp"
Oct 30 19:39:15.534493 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:15.534462    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd"
Oct 30 19:39:15.535005 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:15.534939    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd"
Oct 30 19:39:15.535643 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:15.535621    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd"
Oct 30 19:39:17.573002 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:17.572969    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ae63bf-56c67d7d7b-mgtt5"]
Oct 30 19:39:17.581974 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:17.581941    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ae63bf-76d99cc4d7-fdtz4"]
Oct 30 19:39:18.607066 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:18.607029    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-579ddb/test-model-579ddb-kserve-69c7664c5c-nllwv"]
Oct 30 19:39:18.608255 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:18.608231    5043 cpu_manager.go:414] "containerMap: removing container" podUID="39cf0453-c5af-4fba-8a9c-5d3ace5dc747" containerName="revoke-keys"
Oct 30 19:39:18.608361 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:18.608258    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39cf0453-c5af-4fba-8a9c-5d3ace5dc747" containerName="revoke-keys"
Oct 30 19:39:18.608361 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:18.608304    5043 cpu_manager.go:414] "containerMap: removing container" podUID="40748d3a-8823-40ee-9425-f5e36a8889d6" containerName="istio-proxy"
Oct 30 19:39:18.608361 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:18.608312    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40748d3a-8823-40ee-9425-f5e36a8889d6" containerName="istio-proxy"
Oct 30 19:39:18.608555 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:18.608539    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="40748d3a-8823-40ee-9425-f5e36a8889d6" containerName="istio-proxy"
Oct 30 19:39:18.608637 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:18.608569    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="40748d3a-8823-40ee-9425-f5e36a8889d6" containerName="istio-proxy"
Oct 30 19:39:18.610708 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:18.610685    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="39cf0453-c5af-4fba-8a9c-5d3ace5dc747" containerName="revoke-keys"
Oct 30 19:39:18.616908 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:18.616884    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-579ddb/test-model-579ddb-kserve-69c7664c5c-nllwv"
Oct 30 19:39:18.624518 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:18.624498    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-579ddb\"/\"openshift-service-ca.crt\""
Oct 30 19:39:18.628449 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:18.628420    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-579ddb\"/\"kube-root-ca.crt\""
Oct 30 19:39:18.629053 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:18.629035    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-579ddb\"/\"test-model-579ddb-kserve-self-signed-certs\""
Oct 30 19:39:18.630522 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:18.630500    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-579ddb\"/\"default-dockercfg-fvxpv\""
Oct 30 19:39:18.636362 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:18.636337    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-579ddb/test-model-579ddb-kserve-69c7664c5c-nllwv"]
Oct 30 19:39:18.713222 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:18.713171    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb491571-bc76-4a0b-b931-df9437d7a5bd-dshm\") pod \"test-model-579ddb-kserve-69c7664c5c-nllwv\" (UID: \"cb491571-bc76-4a0b-b931-df9437d7a5bd\") " pod="ai-tenant-e2e-inf-a-579ddb/test-model-579ddb-kserve-69c7664c5c-nllwv"
Oct 30 19:39:18.713404 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:18.713270    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb491571-bc76-4a0b-b931-df9437d7a5bd-tls-certs\") pod \"test-model-579ddb-kserve-69c7664c5c-nllwv\" (UID: \"cb491571-bc76-4a0b-b931-df9437d7a5bd\") " pod="ai-tenant-e2e-inf-a-579ddb/test-model-579ddb-kserve-69c7664c5c-nllwv"
Oct 30 19:39:18.713404 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:18.713337    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb491571-bc76-4a0b-b931-df9437d7a5bd-model-cache\") pod \"test-model-579ddb-kserve-69c7664c5c-nllwv\" (UID: \"cb491571-bc76-4a0b-b931-df9437d7a5bd\") " pod="ai-tenant-e2e-inf-a-579ddb/test-model-579ddb-kserve-69c7664c5c-nllwv"
Oct 30 19:39:18.713404 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:18.713397    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trlzx\" (UniqueName: \"kubernetes.io/projected/cb491571-bc76-4a0b-b931-df9437d7a5bd-kube-api-access-trlzx\") pod \"test-model-579ddb-kserve-69c7664c5c-nllwv\" (UID: \"cb491571-bc76-4a0b-b931-df9437d7a5bd\") " pod="ai-tenant-e2e-inf-a-579ddb/test-model-579ddb-kserve-69c7664c5c-nllwv"
Oct 30 19:39:18.713613 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:18.713583    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb491571-bc76-4a0b-b931-df9437d7a5bd-home\") pod \"test-model-579ddb-kserve-69c7664c5c-nllwv\" (UID: \"cb491571-bc76-4a0b-b931-df9437d7a5bd\") " pod="ai-tenant-e2e-inf-a-579ddb/test-model-579ddb-kserve-69c7664c5c-nllwv"
Oct 30 19:39:18.713660 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:18.713612    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb491571-bc76-4a0b-b931-df9437d7a5bd-tmp-dir\") pod \"test-model-579ddb-kserve-69c7664c5c-nllwv\" (UID: \"cb491571-bc76-4a0b-b931-df9437d7a5bd\") " pod="ai-tenant-e2e-inf-a-579ddb/test-model-579ddb-kserve-69c7664c5c-nllwv"
Oct 30 19:39:18.823762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:18.819595    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb491571-bc76-4a0b-b931-df9437d7a5bd-model-cache\") pod \"test-model-579ddb-kserve-69c7664c5c-nllwv\" (UID: \"cb491571-bc76-4a0b-b931-df9437d7a5bd\") " pod="ai-tenant-e2e-inf-a-579ddb/test-model-579ddb-kserve-69c7664c5c-nllwv"
Oct 30 19:39:18.848115 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:18.848082    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trlzx\" (UniqueName: \"kubernetes.io/projected/cb491571-bc76-4a0b-b931-df9437d7a5bd-kube-api-access-trlzx\") pod \"test-model-579ddb-kserve-69c7664c5c-nllwv\" (UID: \"cb491571-bc76-4a0b-b931-df9437d7a5bd\") " pod="ai-tenant-e2e-inf-a-579ddb/test-model-579ddb-kserve-69c7664c5c-nllwv"
Oct 30 19:39:18.848550 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:18.848526    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb491571-bc76-4a0b-b931-df9437d7a5bd-home\") pod \"test-model-579ddb-kserve-69c7664c5c-nllwv\" (UID: \"cb491571-bc76-4a0b-b931-df9437d7a5bd\") " pod="ai-tenant-e2e-inf-a-579ddb/test-model-579ddb-kserve-69c7664c5c-nllwv"
Oct 30 19:39:18.848710 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:18.848693    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb491571-bc76-4a0b-b931-df9437d7a5bd-tmp-dir\") pod \"test-model-579ddb-kserve-69c7664c5c-nllwv\" (UID: \"cb491571-bc76-4a0b-b931-df9437d7a5bd\") " pod="ai-tenant-e2e-inf-a-579ddb/test-model-579ddb-kserve-69c7664c5c-nllwv"
Oct 30 19:39:18.852915 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:18.852886    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb491571-bc76-4a0b-b931-df9437d7a5bd-home\") pod \"test-model-579ddb-kserve-69c7664c5c-nllwv\" (UID: \"cb491571-bc76-4a0b-b931-df9437d7a5bd\") " pod="ai-tenant-e2e-inf-a-579ddb/test-model-579ddb-kserve-69c7664c5c-nllwv"
Oct 30 19:39:18.870275 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:18.853081    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb491571-bc76-4a0b-b931-df9437d7a5bd-dshm\") pod \"test-model-579ddb-kserve-69c7664c5c-nllwv\" (UID: \"cb491571-bc76-4a0b-b931-df9437d7a5bd\") " pod="ai-tenant-e2e-inf-a-579ddb/test-model-579ddb-kserve-69c7664c5c-nllwv"
Oct 30 19:39:18.870833 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:18.870549    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb491571-bc76-4a0b-b931-df9437d7a5bd-tls-certs\") pod \"test-model-579ddb-kserve-69c7664c5c-nllwv\" (UID: \"cb491571-bc76-4a0b-b931-df9437d7a5bd\") " pod="ai-tenant-e2e-inf-a-579ddb/test-model-579ddb-kserve-69c7664c5c-nllwv"
Oct 30 19:39:18.871574 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:18.840433    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb491571-bc76-4a0b-b931-df9437d7a5bd-model-cache\") pod \"test-model-579ddb-kserve-69c7664c5c-nllwv\" (UID: \"cb491571-bc76-4a0b-b931-df9437d7a5bd\") " pod="ai-tenant-e2e-inf-a-579ddb/test-model-579ddb-kserve-69c7664c5c-nllwv"
Oct 30 19:39:18.871760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:18.870152    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb491571-bc76-4a0b-b931-df9437d7a5bd-dshm\") pod \"test-model-579ddb-kserve-69c7664c5c-nllwv\" (UID: \"cb491571-bc76-4a0b-b931-df9437d7a5bd\") " pod="ai-tenant-e2e-inf-a-579ddb/test-model-579ddb-kserve-69c7664c5c-nllwv"
Oct 30 19:39:18.871875 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:18.853312    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb491571-bc76-4a0b-b931-df9437d7a5bd-tmp-dir\") pod \"test-model-579ddb-kserve-69c7664c5c-nllwv\" (UID: \"cb491571-bc76-4a0b-b931-df9437d7a5bd\") " pod="ai-tenant-e2e-inf-a-579ddb/test-model-579ddb-kserve-69c7664c5c-nllwv"
Oct 30 19:39:18.907852 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:18.887447    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb491571-bc76-4a0b-b931-df9437d7a5bd-tls-certs\") pod \"test-model-579ddb-kserve-69c7664c5c-nllwv\" (UID: \"cb491571-bc76-4a0b-b931-df9437d7a5bd\") " pod="ai-tenant-e2e-inf-a-579ddb/test-model-579ddb-kserve-69c7664c5c-nllwv"
Oct 30 19:39:18.943799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:18.942785    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trlzx\" (UniqueName: \"kubernetes.io/projected/cb491571-bc76-4a0b-b931-df9437d7a5bd-kube-api-access-trlzx\") pod \"test-model-579ddb-kserve-69c7664c5c-nllwv\" (UID: \"cb491571-bc76-4a0b-b931-df9437d7a5bd\") " pod="ai-tenant-e2e-inf-a-579ddb/test-model-579ddb-kserve-69c7664c5c-nllwv"
Oct 30 19:39:19.235837 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:19.235796    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-579ddb/test-model-579ddb-kserve-69c7664c5c-nllwv"
Oct 30 19:39:19.595607 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:39:19.595569    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb491571_bc76_4a0b_b931_df9437d7a5bd.slice/crio-f99c04669e5751f8ca766e9f4a41be392c77fd4b3404e003725fe21717465c15 WatchSource:0}: Error finding container f99c04669e5751f8ca766e9f4a41be392c77fd4b3404e003725fe21717465c15: Status 404 returned error can't find the container with id f99c04669e5751f8ca766e9f4a41be392c77fd4b3404e003725fe21717465c15
Oct 30 19:39:19.599236 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:19.599214    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-579ddb/test-model-579ddb-kserve-69c7664c5c-nllwv"]
Oct 30 19:39:20.536575 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:20.536538    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-579ddb/test-model-579ddb-kserve-69c7664c5c-nllwv" event={"ID":"cb491571-bc76-4a0b-b931-df9437d7a5bd","Type":"ContainerStarted","Data":"61f24be4ad43e4a8745a334f4b6e54bcce10b91fcdd50b69ed0e308b31e25e89"}
Oct 30 19:39:20.537102 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:20.536590    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-579ddb/test-model-579ddb-kserve-69c7664c5c-nllwv" event={"ID":"cb491571-bc76-4a0b-b931-df9437d7a5bd","Type":"ContainerStarted","Data":"f99c04669e5751f8ca766e9f4a41be392c77fd4b3404e003725fe21717465c15"}
Oct 30 19:39:20.559257 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:20.559189    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-579ddb/test-model-579ddb-kserve-69c7664c5c-nllwv" podStartSLOduration=2.55916691 podStartE2EDuration="2.55916691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:20.554295801 +0000 UTC m=+1380.065355768" watchObservedRunningTime="2026-10-30 19:39:20.55916691 +0000 UTC m=+1380.070226876"
Oct 30 19:39:21.031092 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:21.031054    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8c67e12-75v4f"]
Oct 30 19:39:21.035860 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:21.035825    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8c67e12-75v4f"]
Oct 30 19:39:21.236627 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:21.236588    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-579ddb/test-model-579ddb-kserve-69c7664c5c-nllwv"
Oct 30 19:39:22.574114 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:22.574061    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-ae63bf-56c67d7d7b-mgtt5" podUID="9a040422-5cca-40da-87f7-f5451e7e4b5a" containerName="payload-processing" containerID="cri-o://f2c8ddca63164304aed8bd4e65f71f4b9689d506d776579460dee070bf0e9306" gracePeriod=25
Oct 30 19:39:22.582831 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:22.582787    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ae63bf-76d99cc4d7-fdtz4" podUID="2c907135-5b39-45a4-b7b5-c54336e1c1a1" containerName="payload-pre-processing" containerID="cri-o://aaeaa38d81cd644c56300b7336c52716102b1524abfd8701c28bc4ef9ad002d1" gracePeriod=25
Oct 30 19:39:22.650762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:22.650384    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98f1f260-d834-4c0f-a2b2-89817b15f1e9" path="/var/lib/kubelet/pods/98f1f260-d834-4c0f-a2b2-89817b15f1e9/volumes"
Oct 30 19:39:23.954277 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:23.954245    5043 scope.go:122] "RemoveContainer" containerID="f5dd80eab41e12d10a569be0d78c28bb4ebd90b8ce71880650c50736fc81e279"
Oct 30 19:39:24.978912 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:39:24.978876    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:39:26.096881 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:26.096831    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fd3f959-sv82j"]
Oct 30 19:39:26.098122 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:26.098097    5043 cpu_manager.go:414] "containerMap: removing container" podUID="40748d3a-8823-40ee-9425-f5e36a8889d6" containerName="istio-proxy"
Oct 30 19:39:26.098122 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:26.098123    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40748d3a-8823-40ee-9425-f5e36a8889d6" containerName="istio-proxy"
Oct 30 19:39:26.104187 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:26.104166    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6970fe37-h6trh"]
Oct 30 19:39:26.104338 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:26.104314    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd3f959-sv82j"
Oct 30 19:39:26.109073 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:26.109027    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6970fe37-h6trh"
Oct 30 19:39:26.115779 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:26.114062    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:39:26.115779 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:26.114222    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:39:26.115779 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:26.114313    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:39:26.115779 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:26.114378    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:39:26.115779 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:26.115548    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fd3f959-sv82j"]
Oct 30 19:39:26.121438 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:26.121410    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6970fe37-h6trh"]
Oct 30 19:39:26.393582 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:26.393469    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98988463-cb27-4b5d-8e96-bd118b70a212-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fd3f959-sv82j\" (UID: \"98988463-cb27-4b5d-8e96-bd118b70a212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd3f959-sv82j"
Oct 30 19:39:26.393902 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:26.393883    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qgph\" (UniqueName: \"kubernetes.io/projected/98988463-cb27-4b5d-8e96-bd118b70a212-kube-api-access-9qgph\") pod \"maas-api-revoke-sub-4fd3f959-sv82j\" (UID: \"98988463-cb27-4b5d-8e96-bd118b70a212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd3f959-sv82j"
Oct 30 19:39:26.394139 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:26.394124    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c96f23c-c4db-409e-acde-e723d8fe248c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6970fe37-h6trh\" (UID: \"9c96f23c-c4db-409e-acde-e723d8fe248c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6970fe37-h6trh"
Oct 30 19:39:26.394328 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:26.394312    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98988463-cb27-4b5d-8e96-bd118b70a212-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fd3f959-sv82j\" (UID: \"98988463-cb27-4b5d-8e96-bd118b70a212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd3f959-sv82j"
Oct 30 19:39:26.394465 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:26.394452    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c96f23c-c4db-409e-acde-e723d8fe248c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6970fe37-h6trh\" (UID: \"9c96f23c-c4db-409e-acde-e723d8fe248c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6970fe37-h6trh"
Oct 30 19:39:26.394562 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:26.394550    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4zcq\" (UniqueName: \"kubernetes.io/projected/9c96f23c-c4db-409e-acde-e723d8fe248c-kube-api-access-w4zcq\") pod \"maas-api-revoke-sub-6970fe37-h6trh\" (UID: \"9c96f23c-c4db-409e-acde-e723d8fe248c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6970fe37-h6trh"
Oct 30 19:39:26.496851 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:26.496809    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c96f23c-c4db-409e-acde-e723d8fe248c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6970fe37-h6trh\" (UID: \"9c96f23c-c4db-409e-acde-e723d8fe248c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6970fe37-h6trh"
Oct 30 19:39:26.497049 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:26.496922    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98988463-cb27-4b5d-8e96-bd118b70a212-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fd3f959-sv82j\" (UID: \"98988463-cb27-4b5d-8e96-bd118b70a212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd3f959-sv82j"
Oct 30 19:39:26.497049 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:26.496979    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c96f23c-c4db-409e-acde-e723d8fe248c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6970fe37-h6trh\" (UID: \"9c96f23c-c4db-409e-acde-e723d8fe248c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6970fe37-h6trh"
Oct 30 19:39:26.497178 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:26.497100    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4zcq\" (UniqueName: \"kubernetes.io/projected/9c96f23c-c4db-409e-acde-e723d8fe248c-kube-api-access-w4zcq\") pod \"maas-api-revoke-sub-6970fe37-h6trh\" (UID: \"9c96f23c-c4db-409e-acde-e723d8fe248c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6970fe37-h6trh"
Oct 30 19:39:26.497424 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:26.497393    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98988463-cb27-4b5d-8e96-bd118b70a212-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fd3f959-sv82j\" (UID: \"98988463-cb27-4b5d-8e96-bd118b70a212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd3f959-sv82j"
Oct 30 19:39:26.497519 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:26.497465    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qgph\" (UniqueName: \"kubernetes.io/projected/98988463-cb27-4b5d-8e96-bd118b70a212-kube-api-access-9qgph\") pod \"maas-api-revoke-sub-4fd3f959-sv82j\" (UID: \"98988463-cb27-4b5d-8e96-bd118b70a212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd3f959-sv82j"
Oct 30 19:39:26.497712 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:26.497686    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c96f23c-c4db-409e-acde-e723d8fe248c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6970fe37-h6trh\" (UID: \"9c96f23c-c4db-409e-acde-e723d8fe248c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6970fe37-h6trh"
Oct 30 19:39:26.498084 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:26.498011    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98988463-cb27-4b5d-8e96-bd118b70a212-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fd3f959-sv82j\" (UID: \"98988463-cb27-4b5d-8e96-bd118b70a212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd3f959-sv82j"
Oct 30 19:39:26.499711 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:26.499683    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c96f23c-c4db-409e-acde-e723d8fe248c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6970fe37-h6trh\" (UID: \"9c96f23c-c4db-409e-acde-e723d8fe248c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6970fe37-h6trh"
Oct 30 19:39:26.500062 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:26.500042    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98988463-cb27-4b5d-8e96-bd118b70a212-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fd3f959-sv82j\" (UID: \"98988463-cb27-4b5d-8e96-bd118b70a212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd3f959-sv82j"
Oct 30 19:39:26.506831 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:26.506811    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4zcq\" (UniqueName: \"kubernetes.io/projected/9c96f23c-c4db-409e-acde-e723d8fe248c-kube-api-access-w4zcq\") pod \"maas-api-revoke-sub-6970fe37-h6trh\" (UID: \"9c96f23c-c4db-409e-acde-e723d8fe248c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6970fe37-h6trh"
Oct 30 19:39:26.507911 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:26.507892    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qgph\" (UniqueName: \"kubernetes.io/projected/98988463-cb27-4b5d-8e96-bd118b70a212-kube-api-access-9qgph\") pod \"maas-api-revoke-sub-4fd3f959-sv82j\" (UID: \"98988463-cb27-4b5d-8e96-bd118b70a212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd3f959-sv82j"
Oct 30 19:39:26.725420 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:26.725385    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd3f959-sv82j"
Oct 30 19:39:26.777702 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:26.777671    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6970fe37-h6trh"
Oct 30 19:39:27.050142 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:39:27.050114    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98988463_cb27_4b5d_8e96_bd118b70a212.slice/crio-87cf1732511b1070ab4415f71a59eeddb175e5551487ee75ed24e5a582d5aa5a WatchSource:0}: Error finding container 87cf1732511b1070ab4415f71a59eeddb175e5551487ee75ed24e5a582d5aa5a: Status 404 returned error can't find the container with id 87cf1732511b1070ab4415f71a59eeddb175e5551487ee75ed24e5a582d5aa5a
Oct 30 19:39:27.050871 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.050850    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fd3f959-sv82j"]
Oct 30 19:39:27.106276 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:39:27.106246    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c96f23c_c4db_409e_acde_e723d8fe248c.slice/crio-af344957d6f73945c3c67e981ea4806d93a5eb4ddbd5e3c95a3cccc0aec0d0d4 WatchSource:0}: Error finding container af344957d6f73945c3c67e981ea4806d93a5eb4ddbd5e3c95a3cccc0aec0d0d4: Status 404 returned error can't find the container with id af344957d6f73945c3c67e981ea4806d93a5eb4ddbd5e3c95a3cccc0aec0d0d4
Oct 30 19:39:27.106276 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.106270    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6970fe37-h6trh"]
Oct 30 19:39:27.235441 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.235402    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" containerID="cri-o://e748252e2dd735b0c75008dc5053528813b53df641e82f864c4785fa027f86d9" gracePeriod=45
Oct 30 19:39:27.241697 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.241653    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" containerID="cri-o://df48a93b78fe9651bdd1f4ecea80c1b57e74362edcc9e9c789bf3de4a9ed8ac3" gracePeriod=45
Oct 30 19:39:27.276111 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.276076    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": dial tcp 10.132.0.97:8000: connect: connection refused"
Oct 30 19:39:27.364243 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.364166    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main" containerID="cri-o://bc08db6b06d31d1745de943fee4254d60cd224ef621a759c01744954b6518bb5" gracePeriod=45
Oct 30 19:39:27.625372 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.625289    5043 generic.go:356] "Generic (PLEG): container finished" podID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerID="bc08db6b06d31d1745de943fee4254d60cd224ef621a759c01744954b6518bb5" exitCode=0
Oct 30 19:39:27.625523 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.625486    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" event={"ID":"96b55519-545c-4336-8bb1-6415d1fdf7f8","Type":"ContainerDied","Data":"bc08db6b06d31d1745de943fee4254d60cd224ef621a759c01744954b6518bb5"}
Oct 30 19:39:27.631206 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.631172    5043 generic.go:356] "Generic (PLEG): container finished" podID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerID="e748252e2dd735b0c75008dc5053528813b53df641e82f864c4785fa027f86d9" exitCode=0
Oct 30 19:39:27.631354 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.631313    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" event={"ID":"8076c40a-54a0-41a2-b1d6-0d0b13c52351","Type":"ContainerDied","Data":"e748252e2dd735b0c75008dc5053528813b53df641e82f864c4785fa027f86d9"}
Oct 30 19:39:27.633779 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.633728    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd3f959-sv82j" event={"ID":"98988463-cb27-4b5d-8e96-bd118b70a212","Type":"ContainerStarted","Data":"87cf1732511b1070ab4415f71a59eeddb175e5551487ee75ed24e5a582d5aa5a"}
Oct 30 19:39:27.635956 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.635932    5043 generic.go:356] "Generic (PLEG): container finished" podID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerID="df48a93b78fe9651bdd1f4ecea80c1b57e74362edcc9e9c789bf3de4a9ed8ac3" exitCode=0
Oct 30 19:39:27.636061 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.636042    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" event={"ID":"db0f2ea4-b059-4f08-8cdd-a27eddaac604","Type":"ContainerDied","Data":"df48a93b78fe9651bdd1f4ecea80c1b57e74362edcc9e9c789bf3de4a9ed8ac3"}
Oct 30 19:39:27.637774 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.637707    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6970fe37-h6trh" event={"ID":"9c96f23c-c4db-409e-acde-e723d8fe248c","Type":"ContainerStarted","Data":"e00cba2e4fd3b7c639bf8502e73385762b9fc059b8f79b4c1ebaa33f06dd908b"}
Oct 30 19:39:27.637774 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.637767    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6970fe37-h6trh" event={"ID":"9c96f23c-c4db-409e-acde-e723d8fe248c","Type":"ContainerStarted","Data":"af344957d6f73945c3c67e981ea4806d93a5eb4ddbd5e3c95a3cccc0aec0d0d4"}
Oct 30 19:39:27.688897 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.688842    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6970fe37-h6trh" podStartSLOduration=1.688823951 podStartE2EDuration="1.688823951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:27.685230527 +0000 UTC m=+1387.196290493" watchObservedRunningTime="2026-10-30 19:39:27.688823951 +0000 UTC m=+1387.199883916"
Oct 30 19:39:27.739680 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.739650    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"
Oct 30 19:39:27.742129 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.742108    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:39:27.811834 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.811800    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d1d256-svs6j"]
Oct 30 19:39:27.812952 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.812924    5043 cpu_manager.go:414] "containerMap: removing container" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main"
Oct 30 19:39:27.812952 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.812951    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main"
Oct 30 19:39:27.813126 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.813033    5043 cpu_manager.go:414] "containerMap: removing container" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main"
Oct 30 19:39:27.813126 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.813042    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main"
Oct 30 19:39:27.813278 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.813250    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" containerName="main"
Oct 30 19:39:27.813333 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.813282    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main"
Oct 30 19:39:27.817194 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.817171    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d1d256-svs6j"
Oct 30 19:39:27.822912 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.821254    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d1d256-svs6j"]
Oct 30 19:39:27.832469 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.832437    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db0f2ea4-b059-4f08-8cdd-a27eddaac604-kube-api-access-26tj6\" (UniqueName: \"kubernetes.io/projected/db0f2ea4-b059-4f08-8cdd-a27eddaac604-kube-api-access-26tj6\") pod \"db0f2ea4-b059-4f08-8cdd-a27eddaac604\" (UID: \"db0f2ea4-b059-4f08-8cdd-a27eddaac604\") "
Oct 30 19:39:27.832624 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.832552    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8076c40a-54a0-41a2-b1d6-0d0b13c52351-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8076c40a-54a0-41a2-b1d6-0d0b13c52351-dshm\") pod \"8076c40a-54a0-41a2-b1d6-0d0b13c52351\" (UID: \"8076c40a-54a0-41a2-b1d6-0d0b13c52351\") "
Oct 30 19:39:27.832701 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.832627    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db0f2ea4-b059-4f08-8cdd-a27eddaac604-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db0f2ea4-b059-4f08-8cdd-a27eddaac604-model-cache\") pod \"db0f2ea4-b059-4f08-8cdd-a27eddaac604\" (UID: \"db0f2ea4-b059-4f08-8cdd-a27eddaac604\") "
Oct 30 19:39:27.832919 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.832724    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8076c40a-54a0-41a2-b1d6-0d0b13c52351-tls-certs\" (UniqueName: \"kubernetes.io/secret/8076c40a-54a0-41a2-b1d6-0d0b13c52351-tls-certs\") pod \"8076c40a-54a0-41a2-b1d6-0d0b13c52351\" (UID: \"8076c40a-54a0-41a2-b1d6-0d0b13c52351\") "
Oct 30 19:39:27.832919 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.832837    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8076c40a-54a0-41a2-b1d6-0d0b13c52351-home\" (UniqueName: \"kubernetes.io/empty-dir/8076c40a-54a0-41a2-b1d6-0d0b13c52351-home\") pod \"8076c40a-54a0-41a2-b1d6-0d0b13c52351\" (UID: \"8076c40a-54a0-41a2-b1d6-0d0b13c52351\") "
Oct 30 19:39:27.833039 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.832944    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db0f2ea4-b059-4f08-8cdd-a27eddaac604-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db0f2ea4-b059-4f08-8cdd-a27eddaac604-tmp-dir\") pod \"db0f2ea4-b059-4f08-8cdd-a27eddaac604\" (UID: \"db0f2ea4-b059-4f08-8cdd-a27eddaac604\") "
Oct 30 19:39:27.833039 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.832982    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db0f2ea4-b059-4f08-8cdd-a27eddaac604-home\" (UniqueName: \"kubernetes.io/empty-dir/db0f2ea4-b059-4f08-8cdd-a27eddaac604-home\") pod \"db0f2ea4-b059-4f08-8cdd-a27eddaac604\" (UID: \"db0f2ea4-b059-4f08-8cdd-a27eddaac604\") "
Oct 30 19:39:27.833148 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.833057    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8076c40a-54a0-41a2-b1d6-0d0b13c52351-kube-api-access-zpw9f\" (UniqueName: \"kubernetes.io/projected/8076c40a-54a0-41a2-b1d6-0d0b13c52351-kube-api-access-zpw9f\") pod \"8076c40a-54a0-41a2-b1d6-0d0b13c52351\" (UID: \"8076c40a-54a0-41a2-b1d6-0d0b13c52351\") "
Oct 30 19:39:27.833148 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.833094    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db0f2ea4-b059-4f08-8cdd-a27eddaac604-dshm\" (UniqueName: \"kubernetes.io/empty-dir/db0f2ea4-b059-4f08-8cdd-a27eddaac604-dshm\") pod \"db0f2ea4-b059-4f08-8cdd-a27eddaac604\" (UID: \"db0f2ea4-b059-4f08-8cdd-a27eddaac604\") "
Oct 30 19:39:27.833148 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.833134    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db0f2ea4-b059-4f08-8cdd-a27eddaac604-tls-certs\" (UniqueName: \"kubernetes.io/secret/db0f2ea4-b059-4f08-8cdd-a27eddaac604-tls-certs\") pod \"db0f2ea4-b059-4f08-8cdd-a27eddaac604\" (UID: \"db0f2ea4-b059-4f08-8cdd-a27eddaac604\") "
Oct 30 19:39:27.833312 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.833170    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8076c40a-54a0-41a2-b1d6-0d0b13c52351-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8076c40a-54a0-41a2-b1d6-0d0b13c52351-tmp-dir\") pod \"8076c40a-54a0-41a2-b1d6-0d0b13c52351\" (UID: \"8076c40a-54a0-41a2-b1d6-0d0b13c52351\") "
Oct 30 19:39:27.833312 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.833214    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8076c40a-54a0-41a2-b1d6-0d0b13c52351-home" pod "8076c40a-54a0-41a2-b1d6-0d0b13c52351" (UID: "8076c40a-54a0-41a2-b1d6-0d0b13c52351"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:27.833312 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.833242    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8076c40a-54a0-41a2-b1d6-0d0b13c52351-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8076c40a-54a0-41a2-b1d6-0d0b13c52351-model-cache\") pod \"8076c40a-54a0-41a2-b1d6-0d0b13c52351\" (UID: \"8076c40a-54a0-41a2-b1d6-0d0b13c52351\") "
Oct 30 19:39:27.833912 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.833573    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad79a09b-6291-4c55-b9a5-cd0f888a1557-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d1d256-svs6j\" (UID: \"ad79a09b-6291-4c55-b9a5-cd0f888a1557\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d1d256-svs6j"
Oct 30 19:39:27.833912 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.833788    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db0f2ea4-b059-4f08-8cdd-a27eddaac604-model-cache" pod "db0f2ea4-b059-4f08-8cdd-a27eddaac604" (UID: "db0f2ea4-b059-4f08-8cdd-a27eddaac604"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:27.835423 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.835047    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8076c40a-54a0-41a2-b1d6-0d0b13c52351-tmp-dir" pod "8076c40a-54a0-41a2-b1d6-0d0b13c52351" (UID: "8076c40a-54a0-41a2-b1d6-0d0b13c52351"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:27.835423 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.835241    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8076c40a-54a0-41a2-b1d6-0d0b13c52351-model-cache" pod "8076c40a-54a0-41a2-b1d6-0d0b13c52351" (UID: "8076c40a-54a0-41a2-b1d6-0d0b13c52351"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:27.835594 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.835498    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x48cm\" (UniqueName: \"kubernetes.io/projected/ad79a09b-6291-4c55-b9a5-cd0f888a1557-kube-api-access-x48cm\") pod \"maas-api-revoke-keys-e2e-api-a-d1d256-svs6j\" (UID: \"ad79a09b-6291-4c55-b9a5-cd0f888a1557\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d1d256-svs6j"
Oct 30 19:39:27.835594 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.835563    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad79a09b-6291-4c55-b9a5-cd0f888a1557-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d1d256-svs6j\" (UID: \"ad79a09b-6291-4c55-b9a5-cd0f888a1557\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d1d256-svs6j"
Oct 30 19:39:27.836008 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.835985    5043 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8076c40a-54a0-41a2-b1d6-0d0b13c52351-tmp-dir\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:27.836101 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.836014    5043 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8076c40a-54a0-41a2-b1d6-0d0b13c52351-model-cache\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:27.836101 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.836032    5043 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db0f2ea4-b059-4f08-8cdd-a27eddaac604-model-cache\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:27.836101 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.836047    5043 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8076c40a-54a0-41a2-b1d6-0d0b13c52351-home\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:27.836306 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.836283    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db0f2ea4-b059-4f08-8cdd-a27eddaac604-home" pod "db0f2ea4-b059-4f08-8cdd-a27eddaac604" (UID: "db0f2ea4-b059-4f08-8cdd-a27eddaac604"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:27.836480 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.836459    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db0f2ea4-b059-4f08-8cdd-a27eddaac604-tmp-dir" pod "db0f2ea4-b059-4f08-8cdd-a27eddaac604" (UID: "db0f2ea4-b059-4f08-8cdd-a27eddaac604"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:27.841714 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.839081    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8076c40a-54a0-41a2-b1d6-0d0b13c52351-tls-certs" pod "8076c40a-54a0-41a2-b1d6-0d0b13c52351" (UID: "8076c40a-54a0-41a2-b1d6-0d0b13c52351"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:39:27.841714 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.841516    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8076c40a-54a0-41a2-b1d6-0d0b13c52351-kube-api-access-zpw9f" pod "8076c40a-54a0-41a2-b1d6-0d0b13c52351" (UID: "8076c40a-54a0-41a2-b1d6-0d0b13c52351"). InnerVolumeSpecName "kube-api-access-zpw9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:27.844476 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.843726    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db0f2ea4-b059-4f08-8cdd-a27eddaac604-dshm" pod "db0f2ea4-b059-4f08-8cdd-a27eddaac604" (UID: "db0f2ea4-b059-4f08-8cdd-a27eddaac604"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:27.844476 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.844467    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db0f2ea4-b059-4f08-8cdd-a27eddaac604-kube-api-access-26tj6" pod "db0f2ea4-b059-4f08-8cdd-a27eddaac604" (UID: "db0f2ea4-b059-4f08-8cdd-a27eddaac604"). InnerVolumeSpecName "kube-api-access-26tj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:27.847061 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.847039    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8076c40a-54a0-41a2-b1d6-0d0b13c52351-dshm" pod "8076c40a-54a0-41a2-b1d6-0d0b13c52351" (UID: "8076c40a-54a0-41a2-b1d6-0d0b13c52351"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:27.847445 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.847415    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db0f2ea4-b059-4f08-8cdd-a27eddaac604-tls-certs" pod "db0f2ea4-b059-4f08-8cdd-a27eddaac604" (UID: "db0f2ea4-b059-4f08-8cdd-a27eddaac604"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:39:27.861793 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.861407    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:39:27.936822 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.936702    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96b55519-545c-4336-8bb1-6415d1fdf7f8-home\" (UniqueName: \"kubernetes.io/empty-dir/96b55519-545c-4336-8bb1-6415d1fdf7f8-home\") pod \"96b55519-545c-4336-8bb1-6415d1fdf7f8\" (UID: \"96b55519-545c-4336-8bb1-6415d1fdf7f8\") "
Oct 30 19:39:27.936998 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.936950    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96b55519-545c-4336-8bb1-6415d1fdf7f8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96b55519-545c-4336-8bb1-6415d1fdf7f8-tmp-dir\") pod \"96b55519-545c-4336-8bb1-6415d1fdf7f8\" (UID: \"96b55519-545c-4336-8bb1-6415d1fdf7f8\") "
Oct 30 19:39:27.936998 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.936987    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96b55519-545c-4336-8bb1-6415d1fdf7f8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/96b55519-545c-4336-8bb1-6415d1fdf7f8-dshm\") pod \"96b55519-545c-4336-8bb1-6415d1fdf7f8\" (UID: \"96b55519-545c-4336-8bb1-6415d1fdf7f8\") "
Oct 30 19:39:27.937119 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.936990    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96b55519-545c-4336-8bb1-6415d1fdf7f8-home" pod "96b55519-545c-4336-8bb1-6415d1fdf7f8" (UID: "96b55519-545c-4336-8bb1-6415d1fdf7f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:27.937179 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.937125    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96b55519-545c-4336-8bb1-6415d1fdf7f8-tmp-dir" pod "96b55519-545c-4336-8bb1-6415d1fdf7f8" (UID: "96b55519-545c-4336-8bb1-6415d1fdf7f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:27.937179 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.937142    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96b55519-545c-4336-8bb1-6415d1fdf7f8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96b55519-545c-4336-8bb1-6415d1fdf7f8-model-cache\") pod \"96b55519-545c-4336-8bb1-6415d1fdf7f8\" (UID: \"96b55519-545c-4336-8bb1-6415d1fdf7f8\") "
Oct 30 19:39:27.937284 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.937225    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96b55519-545c-4336-8bb1-6415d1fdf7f8-kube-api-access-7w7b4\" (UniqueName: \"kubernetes.io/projected/96b55519-545c-4336-8bb1-6415d1fdf7f8-kube-api-access-7w7b4\") pod \"96b55519-545c-4336-8bb1-6415d1fdf7f8\" (UID: \"96b55519-545c-4336-8bb1-6415d1fdf7f8\") "
Oct 30 19:39:27.937339 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.937303    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96b55519-545c-4336-8bb1-6415d1fdf7f8-tls-certs\" (UniqueName: \"kubernetes.io/secret/96b55519-545c-4336-8bb1-6415d1fdf7f8-tls-certs\") pod \"96b55519-545c-4336-8bb1-6415d1fdf7f8\" (UID: \"96b55519-545c-4336-8bb1-6415d1fdf7f8\") "
Oct 30 19:39:27.937489 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.937462    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96b55519-545c-4336-8bb1-6415d1fdf7f8-model-cache" pod "96b55519-545c-4336-8bb1-6415d1fdf7f8" (UID: "96b55519-545c-4336-8bb1-6415d1fdf7f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:27.937698 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.937673    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad79a09b-6291-4c55-b9a5-cd0f888a1557-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d1d256-svs6j\" (UID: \"ad79a09b-6291-4c55-b9a5-cd0f888a1557\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d1d256-svs6j"
Oct 30 19:39:27.937873 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.937853    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x48cm\" (UniqueName: \"kubernetes.io/projected/ad79a09b-6291-4c55-b9a5-cd0f888a1557-kube-api-access-x48cm\") pod \"maas-api-revoke-keys-e2e-api-a-d1d256-svs6j\" (UID: \"ad79a09b-6291-4c55-b9a5-cd0f888a1557\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d1d256-svs6j"
Oct 30 19:39:27.937982 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.937917    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad79a09b-6291-4c55-b9a5-cd0f888a1557-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d1d256-svs6j\" (UID: \"ad79a09b-6291-4c55-b9a5-cd0f888a1557\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d1d256-svs6j"
Oct 30 19:39:27.938348 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.938328    5043 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db0f2ea4-b059-4f08-8cdd-a27eddaac604-tmp-dir\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:27.938469 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.938358    5043 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db0f2ea4-b059-4f08-8cdd-a27eddaac604-home\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:27.938469 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.938373    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpw9f\" (UniqueName: \"kubernetes.io/projected/8076c40a-54a0-41a2-b1d6-0d0b13c52351-kube-api-access-zpw9f\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:27.938469 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.938387    5043 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96b55519-545c-4336-8bb1-6415d1fdf7f8-model-cache\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:27.938469 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.938402    5043 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db0f2ea4-b059-4f08-8cdd-a27eddaac604-dshm\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:27.938469 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.938415    5043 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db0f2ea4-b059-4f08-8cdd-a27eddaac604-tls-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:27.938469 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.938428    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26tj6\" (UniqueName: \"kubernetes.io/projected/db0f2ea4-b059-4f08-8cdd-a27eddaac604-kube-api-access-26tj6\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:27.938469 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.938442    5043 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8076c40a-54a0-41a2-b1d6-0d0b13c52351-dshm\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:27.938469 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.938455    5043 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96b55519-545c-4336-8bb1-6415d1fdf7f8-home\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:27.938469 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.938467    5043 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8076c40a-54a0-41a2-b1d6-0d0b13c52351-tls-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:27.939109 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.938481    5043 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96b55519-545c-4336-8bb1-6415d1fdf7f8-tmp-dir\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:27.939109 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.938710    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad79a09b-6291-4c55-b9a5-cd0f888a1557-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d1d256-svs6j\" (UID: \"ad79a09b-6291-4c55-b9a5-cd0f888a1557\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d1d256-svs6j"
Oct 30 19:39:27.940026 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.939992    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96b55519-545c-4336-8bb1-6415d1fdf7f8-dshm" pod "96b55519-545c-4336-8bb1-6415d1fdf7f8" (UID: "96b55519-545c-4336-8bb1-6415d1fdf7f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:27.940252 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.940229    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96b55519-545c-4336-8bb1-6415d1fdf7f8-tls-certs" pod "96b55519-545c-4336-8bb1-6415d1fdf7f8" (UID: "96b55519-545c-4336-8bb1-6415d1fdf7f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:39:27.941216 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.941192    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad79a09b-6291-4c55-b9a5-cd0f888a1557-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d1d256-svs6j\" (UID: \"ad79a09b-6291-4c55-b9a5-cd0f888a1557\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d1d256-svs6j"
Oct 30 19:39:27.942153 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.942112    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96b55519-545c-4336-8bb1-6415d1fdf7f8-kube-api-access-7w7b4" pod "96b55519-545c-4336-8bb1-6415d1fdf7f8" (UID: "96b55519-545c-4336-8bb1-6415d1fdf7f8"). InnerVolumeSpecName "kube-api-access-7w7b4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:27.947065 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:27.947037    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x48cm\" (UniqueName: \"kubernetes.io/projected/ad79a09b-6291-4c55-b9a5-cd0f888a1557-kube-api-access-x48cm\") pod \"maas-api-revoke-keys-e2e-api-a-d1d256-svs6j\" (UID: \"ad79a09b-6291-4c55-b9a5-cd0f888a1557\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d1d256-svs6j"
Oct 30 19:39:28.040355 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:28.040325    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w7b4\" (UniqueName: \"kubernetes.io/projected/96b55519-545c-4336-8bb1-6415d1fdf7f8-kube-api-access-7w7b4\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:28.040355 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:28.040350    5043 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96b55519-545c-4336-8bb1-6415d1fdf7f8-tls-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:28.040566 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:28.040370    5043 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96b55519-545c-4336-8bb1-6415d1fdf7f8-dshm\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:28.146577 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:28.146541    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d1d256-svs6j"
Oct 30 19:39:28.437436 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:39:28.437401    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad79a09b_6291_4c55_b9a5_cd0f888a1557.slice/crio-481e4df1d29b24f6143a7952dcce934cea44c9969431c061a075e67d2ec62f28 WatchSource:0}: Error finding container 481e4df1d29b24f6143a7952dcce934cea44c9969431c061a075e67d2ec62f28: Status 404 returned error can't find the container with id 481e4df1d29b24f6143a7952dcce934cea44c9969431c061a075e67d2ec62f28
Oct 30 19:39:28.438576 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:28.438554    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d1d256-svs6j"]
Oct 30 19:39:28.506495 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:28.506455    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ae63bf-76d99cc4d7-fdtz4" podUID="2c907135-5b39-45a4-b7b5-c54336e1c1a1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:39:28.506655 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:28.506601    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-ae63bf-56c67d7d7b-mgtt5" podUID="9a040422-5cca-40da-87f7-f5451e7e4b5a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:39:28.640945 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:28.640901    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:28.649846 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:28.649819    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"
Oct 30 19:39:28.650000 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:28.649842    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4" event={"ID":"8076c40a-54a0-41a2-b1d6-0d0b13c52351","Type":"ContainerDied","Data":"9808062374c1814473ec017fddcd97df2aca89716c8415446aebaaee7ec9f2fa"}
Oct 30 19:39:28.650000 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:28.649891    5043 scope.go:122] "RemoveContainer" containerID="e748252e2dd735b0c75008dc5053528813b53df641e82f864c4785fa027f86d9"
Oct 30 19:39:28.652473 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:28.652351    5043 generic.go:356] "Generic (PLEG): container finished" podID="98988463-cb27-4b5d-8e96-bd118b70a212" containerID="5478b7d677c61fc946a0dd6efdfd3928e3e7ceba65a0285d403565d98d8b38a7" exitCode=0
Oct 30 19:39:28.652473 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:28.652391    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd3f959-sv82j" event={"ID":"98988463-cb27-4b5d-8e96-bd118b70a212","Type":"ContainerDied","Data":"5478b7d677c61fc946a0dd6efdfd3928e3e7ceba65a0285d403565d98d8b38a7"}
Oct 30 19:39:28.654691 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:28.654664    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb" event={"ID":"db0f2ea4-b059-4f08-8cdd-a27eddaac604","Type":"ContainerDied","Data":"5b5e9accbd4bf49065bef0d541a817db9cdc007aa07755297df3b94b6af3beb3"}
Oct 30 19:39:28.654884 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:28.654868    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"
Oct 30 19:39:28.665178 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:28.665150    5043 generic.go:356] "Generic (PLEG): container finished" podID="9c96f23c-c4db-409e-acde-e723d8fe248c" containerID="e00cba2e4fd3b7c639bf8502e73385762b9fc059b8f79b4c1ebaa33f06dd908b" exitCode=0
Oct 30 19:39:28.665307 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:28.665287    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6970fe37-h6trh" event={"ID":"9c96f23c-c4db-409e-acde-e723d8fe248c","Type":"ContainerDied","Data":"e00cba2e4fd3b7c639bf8502e73385762b9fc059b8f79b4c1ebaa33f06dd908b"}
Oct 30 19:39:28.684822 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:28.675095    5043 scope.go:122] "RemoveContainer" containerID="df48a93b78fe9651bdd1f4ecea80c1b57e74362edcc9e9c789bf3de4a9ed8ac3"
Oct 30 19:39:28.730568 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:28.730456    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d1d256-svs6j" event={"ID":"ad79a09b-6291-4c55-b9a5-cd0f888a1557","Type":"ContainerStarted","Data":"481e4df1d29b24f6143a7952dcce934cea44c9969431c061a075e67d2ec62f28"}
Oct 30 19:39:28.750793 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:28.747162    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr" event={"ID":"96b55519-545c-4336-8bb1-6415d1fdf7f8","Type":"ContainerDied","Data":"558d4149c70ff8bec4fe58f7c7e3e32f145cc11b612416bcc009e72410320f17"}
Oct 30 19:39:28.750793 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:28.747288    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"
Oct 30 19:39:28.774538 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:28.774504    5043 scope.go:122] "RemoveContainer" containerID="bc08db6b06d31d1745de943fee4254d60cd224ef621a759c01744954b6518bb5"
Oct 30 19:39:28.800665 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:28.800633    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"]
Oct 30 19:39:28.807052 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:28.807024    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ae63bf/distinct-w0-ae63bf-kserve-769676c696-nszz4"]
Oct 30 19:39:28.812272 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:28.812251    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"]
Oct 30 19:39:28.816489 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:28.816468    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ae63bf/premium-w0-ae63bf-kserve-865c7d559c-shtjb"]
Oct 30 19:39:28.827759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:28.827725    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"]
Oct 30 19:39:28.831544 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:28.831525    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ae63bf/simulator-w0-ae63bf-kserve-7d6dddf4fb-lznqr"]
Oct 30 19:39:29.238011 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:29.236141    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-579ddb/test-model-579ddb-kserve-69c7664c5c-nllwv"
Oct 30 19:39:29.265430 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:29.256169    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-579ddb/test-model-579ddb-kserve-69c7664c5c-nllwv"
Oct 30 19:39:29.758971 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:29.758882    5043 generic.go:356] "Generic (PLEG): container finished" podID="ad79a09b-6291-4c55-b9a5-cd0f888a1557" containerID="f7174cc773f830b1388a2a84feb355e4448d37b651fd88e6f63fccc37a870165" exitCode=0
Oct 30 19:39:29.758971 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:29.758957    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d1d256-svs6j" event={"ID":"ad79a09b-6291-4c55-b9a5-cd0f888a1557","Type":"ContainerDied","Data":"f7174cc773f830b1388a2a84feb355e4448d37b651fd88e6f63fccc37a870165"}
Oct 30 19:39:29.777382 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:29.777354    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-579ddb/test-model-579ddb-kserve-69c7664c5c-nllwv"
Oct 30 19:39:29.845021 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:29.844957    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82012b-xc65q"]
Oct 30 19:39:29.854406 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:29.854058    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82012b-xc65q"]
Oct 30 19:39:29.915349 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:29.915313    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-1e13df/test-model-1e13df-kserve-c8b994f5f-97xvv"]
Oct 30 19:39:29.916331 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:29.916308    5043 cpu_manager.go:414] "containerMap: removing container" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main"
Oct 30 19:39:29.916331 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:29.916331    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main"
Oct 30 19:39:29.916767 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:29.916750    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" containerName="main"
Oct 30 19:39:29.921387 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:29.921357    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1e13df/test-model-1e13df-kserve-c8b994f5f-97xvv"]
Oct 30 19:39:29.921520 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:29.921503    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1e13df/test-model-1e13df-kserve-c8b994f5f-97xvv"
Oct 30 19:39:29.932201 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:29.931400    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1e13df\"/\"default-dockercfg-htbg8\""
Oct 30 19:39:29.932201 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:29.931513    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1e13df\"/\"test-model-1e13df-kserve-self-signed-certs\""
Oct 30 19:39:29.932201 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:29.931941    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1e13df\"/\"kube-root-ca.crt\""
Oct 30 19:39:29.935149 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:29.935132    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1e13df\"/\"openshift-service-ca.crt\""
Oct 30 19:39:30.015340 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.013439    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d3381d2-2f2e-4699-bd0b-300f2355df74-dshm\") pod \"test-model-1e13df-kserve-c8b994f5f-97xvv\" (UID: \"4d3381d2-2f2e-4699-bd0b-300f2355df74\") " pod="ai-tenant-e2e-inf-b-1e13df/test-model-1e13df-kserve-c8b994f5f-97xvv"
Oct 30 19:39:30.015340 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.013477    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d3381d2-2f2e-4699-bd0b-300f2355df74-tmp-dir\") pod \"test-model-1e13df-kserve-c8b994f5f-97xvv\" (UID: \"4d3381d2-2f2e-4699-bd0b-300f2355df74\") " pod="ai-tenant-e2e-inf-b-1e13df/test-model-1e13df-kserve-c8b994f5f-97xvv"
Oct 30 19:39:30.015340 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.013517    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d3381d2-2f2e-4699-bd0b-300f2355df74-model-cache\") pod \"test-model-1e13df-kserve-c8b994f5f-97xvv\" (UID: \"4d3381d2-2f2e-4699-bd0b-300f2355df74\") " pod="ai-tenant-e2e-inf-b-1e13df/test-model-1e13df-kserve-c8b994f5f-97xvv"
Oct 30 19:39:30.015340 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.013574    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47lzd\" (UniqueName: \"kubernetes.io/projected/4d3381d2-2f2e-4699-bd0b-300f2355df74-kube-api-access-47lzd\") pod \"test-model-1e13df-kserve-c8b994f5f-97xvv\" (UID: \"4d3381d2-2f2e-4699-bd0b-300f2355df74\") " pod="ai-tenant-e2e-inf-b-1e13df/test-model-1e13df-kserve-c8b994f5f-97xvv"
Oct 30 19:39:30.015340 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.013616    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d3381d2-2f2e-4699-bd0b-300f2355df74-home\") pod \"test-model-1e13df-kserve-c8b994f5f-97xvv\" (UID: \"4d3381d2-2f2e-4699-bd0b-300f2355df74\") " pod="ai-tenant-e2e-inf-b-1e13df/test-model-1e13df-kserve-c8b994f5f-97xvv"
Oct 30 19:39:30.015340 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.013666    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d3381d2-2f2e-4699-bd0b-300f2355df74-tls-certs\") pod \"test-model-1e13df-kserve-c8b994f5f-97xvv\" (UID: \"4d3381d2-2f2e-4699-bd0b-300f2355df74\") " pod="ai-tenant-e2e-inf-b-1e13df/test-model-1e13df-kserve-c8b994f5f-97xvv"
Oct 30 19:39:30.053267 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.053234    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd3f959-sv82j"
Oct 30 19:39:30.155767 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.146906    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d3381d2-2f2e-4699-bd0b-300f2355df74-dshm\") pod \"test-model-1e13df-kserve-c8b994f5f-97xvv\" (UID: \"4d3381d2-2f2e-4699-bd0b-300f2355df74\") " pod="ai-tenant-e2e-inf-b-1e13df/test-model-1e13df-kserve-c8b994f5f-97xvv"
Oct 30 19:39:30.155767 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.146954    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d3381d2-2f2e-4699-bd0b-300f2355df74-tmp-dir\") pod \"test-model-1e13df-kserve-c8b994f5f-97xvv\" (UID: \"4d3381d2-2f2e-4699-bd0b-300f2355df74\") " pod="ai-tenant-e2e-inf-b-1e13df/test-model-1e13df-kserve-c8b994f5f-97xvv"
Oct 30 19:39:30.155767 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.146992    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d3381d2-2f2e-4699-bd0b-300f2355df74-model-cache\") pod \"test-model-1e13df-kserve-c8b994f5f-97xvv\" (UID: \"4d3381d2-2f2e-4699-bd0b-300f2355df74\") " pod="ai-tenant-e2e-inf-b-1e13df/test-model-1e13df-kserve-c8b994f5f-97xvv"
Oct 30 19:39:30.155767 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.147048    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47lzd\" (UniqueName: \"kubernetes.io/projected/4d3381d2-2f2e-4699-bd0b-300f2355df74-kube-api-access-47lzd\") pod \"test-model-1e13df-kserve-c8b994f5f-97xvv\" (UID: \"4d3381d2-2f2e-4699-bd0b-300f2355df74\") " pod="ai-tenant-e2e-inf-b-1e13df/test-model-1e13df-kserve-c8b994f5f-97xvv"
Oct 30 19:39:30.155767 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.147087    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d3381d2-2f2e-4699-bd0b-300f2355df74-home\") pod \"test-model-1e13df-kserve-c8b994f5f-97xvv\" (UID: \"4d3381d2-2f2e-4699-bd0b-300f2355df74\") " pod="ai-tenant-e2e-inf-b-1e13df/test-model-1e13df-kserve-c8b994f5f-97xvv"
Oct 30 19:39:30.155767 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.147135    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d3381d2-2f2e-4699-bd0b-300f2355df74-tls-certs\") pod \"test-model-1e13df-kserve-c8b994f5f-97xvv\" (UID: \"4d3381d2-2f2e-4699-bd0b-300f2355df74\") " pod="ai-tenant-e2e-inf-b-1e13df/test-model-1e13df-kserve-c8b994f5f-97xvv"
Oct 30 19:39:30.155767 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.154202    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d3381d2-2f2e-4699-bd0b-300f2355df74-tmp-dir\") pod \"test-model-1e13df-kserve-c8b994f5f-97xvv\" (UID: \"4d3381d2-2f2e-4699-bd0b-300f2355df74\") " pod="ai-tenant-e2e-inf-b-1e13df/test-model-1e13df-kserve-c8b994f5f-97xvv"
Oct 30 19:39:30.170774 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.157058    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d3381d2-2f2e-4699-bd0b-300f2355df74-model-cache\") pod \"test-model-1e13df-kserve-c8b994f5f-97xvv\" (UID: \"4d3381d2-2f2e-4699-bd0b-300f2355df74\") " pod="ai-tenant-e2e-inf-b-1e13df/test-model-1e13df-kserve-c8b994f5f-97xvv"
Oct 30 19:39:30.170774 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.162822    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d3381d2-2f2e-4699-bd0b-300f2355df74-home\") pod \"test-model-1e13df-kserve-c8b994f5f-97xvv\" (UID: \"4d3381d2-2f2e-4699-bd0b-300f2355df74\") " pod="ai-tenant-e2e-inf-b-1e13df/test-model-1e13df-kserve-c8b994f5f-97xvv"
Oct 30 19:39:30.172341 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.172313    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6970fe37-h6trh"
Oct 30 19:39:30.178522 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.178369    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47lzd\" (UniqueName: \"kubernetes.io/projected/4d3381d2-2f2e-4699-bd0b-300f2355df74-kube-api-access-47lzd\") pod \"test-model-1e13df-kserve-c8b994f5f-97xvv\" (UID: \"4d3381d2-2f2e-4699-bd0b-300f2355df74\") " pod="ai-tenant-e2e-inf-b-1e13df/test-model-1e13df-kserve-c8b994f5f-97xvv"
Oct 30 19:39:30.186214 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.186157    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d3381d2-2f2e-4699-bd0b-300f2355df74-tls-certs\") pod \"test-model-1e13df-kserve-c8b994f5f-97xvv\" (UID: \"4d3381d2-2f2e-4699-bd0b-300f2355df74\") " pod="ai-tenant-e2e-inf-b-1e13df/test-model-1e13df-kserve-c8b994f5f-97xvv"
Oct 30 19:39:30.201941 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.201912    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d3381d2-2f2e-4699-bd0b-300f2355df74-dshm\") pod \"test-model-1e13df-kserve-c8b994f5f-97xvv\" (UID: \"4d3381d2-2f2e-4699-bd0b-300f2355df74\") " pod="ai-tenant-e2e-inf-b-1e13df/test-model-1e13df-kserve-c8b994f5f-97xvv"
Oct 30 19:39:30.245151 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.245117    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1e13df/test-model-1e13df-kserve-c8b994f5f-97xvv"
Oct 30 19:39:30.257268 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.257142    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98988463-cb27-4b5d-8e96-bd118b70a212-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98988463-cb27-4b5d-8e96-bd118b70a212-maas-api-openshift-ca\") pod \"98988463-cb27-4b5d-8e96-bd118b70a212\" (UID: \"98988463-cb27-4b5d-8e96-bd118b70a212\") "
Oct 30 19:39:30.258016 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.257995    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c96f23c-c4db-409e-acde-e723d8fe248c-kube-api-access-w4zcq\" (UniqueName: \"kubernetes.io/projected/9c96f23c-c4db-409e-acde-e723d8fe248c-kube-api-access-w4zcq\") pod \"9c96f23c-c4db-409e-acde-e723d8fe248c\" (UID: \"9c96f23c-c4db-409e-acde-e723d8fe248c\") "
Oct 30 19:39:30.258534 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.258516    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c96f23c-c4db-409e-acde-e723d8fe248c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c96f23c-c4db-409e-acde-e723d8fe248c-maas-api-openshift-ca\") pod \"9c96f23c-c4db-409e-acde-e723d8fe248c\" (UID: \"9c96f23c-c4db-409e-acde-e723d8fe248c\") "
Oct 30 19:39:30.258772 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.258753    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98988463-cb27-4b5d-8e96-bd118b70a212-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98988463-cb27-4b5d-8e96-bd118b70a212-maas-api-xks-ca\") pod \"98988463-cb27-4b5d-8e96-bd118b70a212\" (UID: \"98988463-cb27-4b5d-8e96-bd118b70a212\") "
Oct 30 19:39:30.258964 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.258951    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c96f23c-c4db-409e-acde-e723d8fe248c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c96f23c-c4db-409e-acde-e723d8fe248c-maas-api-xks-ca\") pod \"9c96f23c-c4db-409e-acde-e723d8fe248c\" (UID: \"9c96f23c-c4db-409e-acde-e723d8fe248c\") "
Oct 30 19:39:30.259135 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.259112    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98988463-cb27-4b5d-8e96-bd118b70a212-kube-api-access-9qgph\" (UniqueName: \"kubernetes.io/projected/98988463-cb27-4b5d-8e96-bd118b70a212-kube-api-access-9qgph\") pod \"98988463-cb27-4b5d-8e96-bd118b70a212\" (UID: \"98988463-cb27-4b5d-8e96-bd118b70a212\") "
Oct 30 19:39:30.260514 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.257927    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98988463-cb27-4b5d-8e96-bd118b70a212-maas-api-openshift-ca" pod "98988463-cb27-4b5d-8e96-bd118b70a212" (UID: "98988463-cb27-4b5d-8e96-bd118b70a212"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:30.260625 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.260477    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c96f23c-c4db-409e-acde-e723d8fe248c-maas-api-openshift-ca" pod "9c96f23c-c4db-409e-acde-e723d8fe248c" (UID: "9c96f23c-c4db-409e-acde-e723d8fe248c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:30.265788 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.265464    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c96f23c-c4db-409e-acde-e723d8fe248c-maas-api-xks-ca" pod "9c96f23c-c4db-409e-acde-e723d8fe248c" (UID: "9c96f23c-c4db-409e-acde-e723d8fe248c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:39:30.283600 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.282687    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98988463-cb27-4b5d-8e96-bd118b70a212-kube-api-access-9qgph" pod "98988463-cb27-4b5d-8e96-bd118b70a212" (UID: "98988463-cb27-4b5d-8e96-bd118b70a212"). InnerVolumeSpecName "kube-api-access-9qgph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:30.290545 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.290508    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98988463-cb27-4b5d-8e96-bd118b70a212-maas-api-xks-ca" pod "98988463-cb27-4b5d-8e96-bd118b70a212" (UID: "98988463-cb27-4b5d-8e96-bd118b70a212"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:39:30.291077 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.291038    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c96f23c-c4db-409e-acde-e723d8fe248c-kube-api-access-w4zcq" pod "9c96f23c-c4db-409e-acde-e723d8fe248c" (UID: "9c96f23c-c4db-409e-acde-e723d8fe248c"). InnerVolumeSpecName "kube-api-access-w4zcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:30.367414 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.367383    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qgph\" (UniqueName: \"kubernetes.io/projected/98988463-cb27-4b5d-8e96-bd118b70a212-kube-api-access-9qgph\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:30.367567 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.367418    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98988463-cb27-4b5d-8e96-bd118b70a212-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:30.367567 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.367437    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4zcq\" (UniqueName: \"kubernetes.io/projected/9c96f23c-c4db-409e-acde-e723d8fe248c-kube-api-access-w4zcq\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:30.367567 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.367450    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c96f23c-c4db-409e-acde-e723d8fe248c-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:30.367567 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.367464    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98988463-cb27-4b5d-8e96-bd118b70a212-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:30.367567 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.367477    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c96f23c-c4db-409e-acde-e723d8fe248c-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:30.571058 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:39:30.571007    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d3381d2_2f2e_4699_bd0b_300f2355df74.slice/crio-dc62d98465e90920b04f81d391135a63869c76a2696f20d91e27d86be0363339 WatchSource:0}: Error finding container dc62d98465e90920b04f81d391135a63869c76a2696f20d91e27d86be0363339: Status 404 returned error can't find the container with id dc62d98465e90920b04f81d391135a63869c76a2696f20d91e27d86be0363339
Oct 30 19:39:30.571540 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.571507    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1e13df/test-model-1e13df-kserve-c8b994f5f-97xvv"]
Oct 30 19:39:30.657838 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.657806    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c267fa9-6d83-4c9e-a9fb-10885801a21f" path="/var/lib/kubelet/pods/5c267fa9-6d83-4c9e-a9fb-10885801a21f/volumes"
Oct 30 19:39:30.658435 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.658400    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8076c40a-54a0-41a2-b1d6-0d0b13c52351" path="/var/lib/kubelet/pods/8076c40a-54a0-41a2-b1d6-0d0b13c52351/volumes"
Oct 30 19:39:30.659068 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.659048    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96b55519-545c-4336-8bb1-6415d1fdf7f8" path="/var/lib/kubelet/pods/96b55519-545c-4336-8bb1-6415d1fdf7f8/volumes"
Oct 30 19:39:30.660048 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.660028    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db0f2ea4-b059-4f08-8cdd-a27eddaac604" path="/var/lib/kubelet/pods/db0f2ea4-b059-4f08-8cdd-a27eddaac604/volumes"
Oct 30 19:39:30.774830 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.774726    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6970fe37-h6trh"
Oct 30 19:39:30.774830 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.774788    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6970fe37-h6trh" event={"ID":"9c96f23c-c4db-409e-acde-e723d8fe248c","Type":"ContainerDied","Data":"af344957d6f73945c3c67e981ea4806d93a5eb4ddbd5e3c95a3cccc0aec0d0d4"}
Oct 30 19:39:30.775099 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.774845    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af344957d6f73945c3c67e981ea4806d93a5eb4ddbd5e3c95a3cccc0aec0d0d4"
Oct 30 19:39:30.777728 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.777298    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1e13df/test-model-1e13df-kserve-c8b994f5f-97xvv" event={"ID":"4d3381d2-2f2e-4699-bd0b-300f2355df74","Type":"ContainerStarted","Data":"4a1aaf8c7a924e98d7d08bde155a26cd5525bdb9e90bce1c19ba28e692dbdf7c"}
Oct 30 19:39:30.777728 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.777333    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1e13df/test-model-1e13df-kserve-c8b994f5f-97xvv" event={"ID":"4d3381d2-2f2e-4699-bd0b-300f2355df74","Type":"ContainerStarted","Data":"dc62d98465e90920b04f81d391135a63869c76a2696f20d91e27d86be0363339"}
Oct 30 19:39:30.779033 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.779009    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd3f959-sv82j" event={"ID":"98988463-cb27-4b5d-8e96-bd118b70a212","Type":"ContainerDied","Data":"87cf1732511b1070ab4415f71a59eeddb175e5551487ee75ed24e5a582d5aa5a"}
Oct 30 19:39:30.779159 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.779047    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd3f959-sv82j"
Oct 30 19:39:30.779159 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.779052    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87cf1732511b1070ab4415f71a59eeddb175e5551487ee75ed24e5a582d5aa5a"
Oct 30 19:39:30.796709 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.796642    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-1e13df/test-model-1e13df-kserve-c8b994f5f-97xvv" podStartSLOduration=1.796624579 podStartE2EDuration="1.796624579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:30.79653041 +0000 UTC m=+1390.307590374" watchObservedRunningTime="2026-10-30 19:39:30.796624579 +0000 UTC m=+1390.307684604"
Oct 30 19:39:30.986535 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:30.986506    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d1d256-svs6j"
Oct 30 19:39:31.080822 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:31.080790    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad79a09b-6291-4c55-b9a5-cd0f888a1557-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad79a09b-6291-4c55-b9a5-cd0f888a1557-maas-api-openshift-ca\") pod \"ad79a09b-6291-4c55-b9a5-cd0f888a1557\" (UID: \"ad79a09b-6291-4c55-b9a5-cd0f888a1557\") "
Oct 30 19:39:31.080822 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:31.080835    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad79a09b-6291-4c55-b9a5-cd0f888a1557-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad79a09b-6291-4c55-b9a5-cd0f888a1557-maas-api-xks-ca\") pod \"ad79a09b-6291-4c55-b9a5-cd0f888a1557\" (UID: \"ad79a09b-6291-4c55-b9a5-cd0f888a1557\") "
Oct 30 19:39:31.081126 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:31.081077    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad79a09b-6291-4c55-b9a5-cd0f888a1557-kube-api-access-x48cm\" (UniqueName: \"kubernetes.io/projected/ad79a09b-6291-4c55-b9a5-cd0f888a1557-kube-api-access-x48cm\") pod \"ad79a09b-6291-4c55-b9a5-cd0f888a1557\" (UID: \"ad79a09b-6291-4c55-b9a5-cd0f888a1557\") "
Oct 30 19:39:31.081237 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:31.081207    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad79a09b-6291-4c55-b9a5-cd0f888a1557-maas-api-openshift-ca" pod "ad79a09b-6291-4c55-b9a5-cd0f888a1557" (UID: "ad79a09b-6291-4c55-b9a5-cd0f888a1557"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:31.081719 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:31.081648    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad79a09b-6291-4c55-b9a5-cd0f888a1557-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:31.083250 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:31.083228    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad79a09b-6291-4c55-b9a5-cd0f888a1557-maas-api-xks-ca" pod "ad79a09b-6291-4c55-b9a5-cd0f888a1557" (UID: "ad79a09b-6291-4c55-b9a5-cd0f888a1557"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:39:31.083400 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:31.083381    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad79a09b-6291-4c55-b9a5-cd0f888a1557-kube-api-access-x48cm" pod "ad79a09b-6291-4c55-b9a5-cd0f888a1557" (UID: "ad79a09b-6291-4c55-b9a5-cd0f888a1557"). InnerVolumeSpecName "kube-api-access-x48cm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:31.183286 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:31.183190    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad79a09b-6291-4c55-b9a5-cd0f888a1557-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:31.183286 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:31.183224    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x48cm\" (UniqueName: \"kubernetes.io/projected/ad79a09b-6291-4c55-b9a5-cd0f888a1557-kube-api-access-x48cm\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:31.246585 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:31.246547    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-1e13df/test-model-1e13df-kserve-c8b994f5f-97xvv"
Oct 30 19:39:31.806713 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:31.806628    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d1d256-svs6j"
Oct 30 19:39:31.806713 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:31.806649    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d1d256-svs6j" event={"ID":"ad79a09b-6291-4c55-b9a5-cd0f888a1557","Type":"ContainerDied","Data":"481e4df1d29b24f6143a7952dcce934cea44c9969431c061a075e67d2ec62f28"}
Oct 30 19:39:31.806713 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:31.806699    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="481e4df1d29b24f6143a7952dcce934cea44c9969431c061a075e67d2ec62f28"
Oct 30 19:39:37.926230 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:37.926193    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d1d256-5b65b66d9d-2djrn"]
Oct 30 19:39:37.973865 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:37.973823    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d1d256-5b889bbcc5-qb7k8"]
Oct 30 19:39:39.409768 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:39.408907    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ae63bf-76d99cc4d7-fdtz4" podUID="2c907135-5b39-45a4-b7b5-c54336e1c1a1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:39:39.435764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:39.410819    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-ae63bf-56c67d7d7b-mgtt5" podUID="9a040422-5cca-40da-87f7-f5451e7e4b5a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:39:39.456630 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:39.446983    5043 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded"
Oct 30 19:39:39.456630 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:39.446850    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:39:39.608764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:39.602797    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r"]
Oct 30 19:39:39.608764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:39.603962    5043 cpu_manager.go:414] "containerMap: removing container" podUID="98988463-cb27-4b5d-8e96-bd118b70a212" containerName="revoke-keys"
Oct 30 19:39:39.608764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:39.603980    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98988463-cb27-4b5d-8e96-bd118b70a212" containerName="revoke-keys"
Oct 30 19:39:39.608764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:39.603992    5043 cpu_manager.go:414] "containerMap: removing container" podUID="ad79a09b-6291-4c55-b9a5-cd0f888a1557" containerName="revoke-keys"
Oct 30 19:39:39.608764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:39.603999    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad79a09b-6291-4c55-b9a5-cd0f888a1557" containerName="revoke-keys"
Oct 30 19:39:39.608764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:39.604023    5043 cpu_manager.go:414] "containerMap: removing container" podUID="9c96f23c-c4db-409e-acde-e723d8fe248c" containerName="revoke-keys"
Oct 30 19:39:39.608764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:39.604030    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c96f23c-c4db-409e-acde-e723d8fe248c" containerName="revoke-keys"
Oct 30 19:39:39.608764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:39.604220    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="98988463-cb27-4b5d-8e96-bd118b70a212" containerName="revoke-keys"
Oct 30 19:39:39.608764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:39.604235    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad79a09b-6291-4c55-b9a5-cd0f888a1557" containerName="revoke-keys"
Oct 30 19:39:39.608764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:39.604246    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c96f23c-c4db-409e-acde-e723d8fe248c" containerName="revoke-keys"
Oct 30 19:39:39.650084 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:39.620691    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r"
Oct 30 19:39:39.650084 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:39.629901    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ac37c8-openshift-default-dockercfg-f7pmb\""
Oct 30 19:39:39.650084 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:39.648345    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r"]
Oct 30 19:39:39.724827 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:39.704258    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b4e8e9-54ac-4461-b99c-a6f0361505b6-workload-socket\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r\" (UID: \"76b4e8e9-54ac-4461-b99c-a6f0361505b6\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r"
Oct 30 19:39:39.726206 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:39.726171    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b4e8e9-54ac-4461-b99c-a6f0361505b6-credential-socket\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r\" (UID: \"76b4e8e9-54ac-4461-b99c-a6f0361505b6\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r"
Oct 30 19:39:39.726557 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:39.726537    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76b4e8e9-54ac-4461-b99c-a6f0361505b6-istio-podinfo\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r\" (UID: \"76b4e8e9-54ac-4461-b99c-a6f0361505b6\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r"
Oct 30 19:39:39.746982 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:39.746935    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76b4e8e9-54ac-4461-b99c-a6f0361505b6-istio-envoy\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r\" (UID: \"76b4e8e9-54ac-4461-b99c-a6f0361505b6\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r"
Oct 30 19:39:39.747266 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:39.747247    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76b4e8e9-54ac-4461-b99c-a6f0361505b6-istiod-ca-cert\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r\" (UID: \"76b4e8e9-54ac-4461-b99c-a6f0361505b6\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r"
Oct 30 19:39:39.747395 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:39.747379    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76b4e8e9-54ac-4461-b99c-a6f0361505b6-istio-data\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r\" (UID: \"76b4e8e9-54ac-4461-b99c-a6f0361505b6\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r"
Oct 30 19:39:39.747521 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:39.747508    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76b4e8e9-54ac-4461-b99c-a6f0361505b6-istio-token\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r\" (UID: \"76b4e8e9-54ac-4461-b99c-a6f0361505b6\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r"
Oct 30 19:39:39.747858 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:39.747838    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76b4e8e9-54ac-4461-b99c-a6f0361505b6-workload-certs\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r\" (UID: \"76b4e8e9-54ac-4461-b99c-a6f0361505b6\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r"
Oct 30 19:39:39.748013 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:39.747998    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64tgh\" (UniqueName: \"kubernetes.io/projected/76b4e8e9-54ac-4461-b99c-a6f0361505b6-kube-api-access-64tgh\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r\" (UID: \"76b4e8e9-54ac-4461-b99c-a6f0361505b6\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r"
Oct 30 19:39:39.835964 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:39.835926    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-156838-ljhgm"]
Oct 30 19:39:39.883761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:39.851660    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76b4e8e9-54ac-4461-b99c-a6f0361505b6-istio-envoy\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r\" (UID: \"76b4e8e9-54ac-4461-b99c-a6f0361505b6\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r"
Oct 30 19:39:39.883761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:39.851713    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76b4e8e9-54ac-4461-b99c-a6f0361505b6-istiod-ca-cert\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r\" (UID: \"76b4e8e9-54ac-4461-b99c-a6f0361505b6\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r"
Oct 30 19:39:39.883761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:39.851760    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76b4e8e9-54ac-4461-b99c-a6f0361505b6-istio-data\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r\" (UID: \"76b4e8e9-54ac-4461-b99c-a6f0361505b6\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r"
Oct 30 19:39:39.883761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:39.851799    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76b4e8e9-54ac-4461-b99c-a6f0361505b6-istio-token\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r\" (UID: \"76b4e8e9-54ac-4461-b99c-a6f0361505b6\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r"
Oct 30 19:39:39.883761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:39.851947    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76b4e8e9-54ac-4461-b99c-a6f0361505b6-workload-certs\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r\" (UID: \"76b4e8e9-54ac-4461-b99c-a6f0361505b6\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r"
Oct 30 19:39:39.883761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:39.851986    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64tgh\" (UniqueName: \"kubernetes.io/projected/76b4e8e9-54ac-4461-b99c-a6f0361505b6-kube-api-access-64tgh\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r\" (UID: \"76b4e8e9-54ac-4461-b99c-a6f0361505b6\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r"
Oct 30 19:39:39.883761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:39.852016    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b4e8e9-54ac-4461-b99c-a6f0361505b6-workload-socket\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r\" (UID: \"76b4e8e9-54ac-4461-b99c-a6f0361505b6\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r"
Oct 30 19:39:39.883761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:39.852041    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b4e8e9-54ac-4461-b99c-a6f0361505b6-credential-socket\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r\" (UID: \"76b4e8e9-54ac-4461-b99c-a6f0361505b6\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r"
Oct 30 19:39:39.883761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:39.852162    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76b4e8e9-54ac-4461-b99c-a6f0361505b6-istio-podinfo\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r\" (UID: \"76b4e8e9-54ac-4461-b99c-a6f0361505b6\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r"
Oct 30 19:39:39.883761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:39.869453    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76b4e8e9-54ac-4461-b99c-a6f0361505b6-workload-certs\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r\" (UID: \"76b4e8e9-54ac-4461-b99c-a6f0361505b6\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r"
Oct 30 19:39:39.913881 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:39.913847    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-156838-ljhgm"]
Oct 30 19:39:39.953763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:39.914085    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b4e8e9-54ac-4461-b99c-a6f0361505b6-workload-socket\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r\" (UID: \"76b4e8e9-54ac-4461-b99c-a6f0361505b6\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r"
Oct 30 19:39:39.984829 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:39.914324    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b4e8e9-54ac-4461-b99c-a6f0361505b6-credential-socket\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r\" (UID: \"76b4e8e9-54ac-4461-b99c-a6f0361505b6\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r"
Oct 30 19:39:39.989765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:39.915001    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-156838-ljhgm"
Oct 30 19:39:40.003771 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:39.990327    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76b4e8e9-54ac-4461-b99c-a6f0361505b6-istiod-ca-cert\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r\" (UID: \"76b4e8e9-54ac-4461-b99c-a6f0361505b6\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r"
Oct 30 19:39:40.096760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:39.990301    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76b4e8e9-54ac-4461-b99c-a6f0361505b6-istio-data\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r\" (UID: \"76b4e8e9-54ac-4461-b99c-a6f0361505b6\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r"
Oct 30 19:39:40.096760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:40.011111    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:39:40.096760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:40.042038    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76b4e8e9-54ac-4461-b99c-a6f0361505b6-istio-envoy\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r\" (UID: \"76b4e8e9-54ac-4461-b99c-a6f0361505b6\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r"
Oct 30 19:39:40.096760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:40.052413    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07ec9d46-74d0-4638-bdbd-67f286deb315-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-156838-ljhgm\" (UID: \"07ec9d46-74d0-4638-bdbd-67f286deb315\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-156838-ljhgm"
Oct 30 19:39:40.096760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:40.061346    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07ec9d46-74d0-4638-bdbd-67f286deb315-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-156838-ljhgm\" (UID: \"07ec9d46-74d0-4638-bdbd-67f286deb315\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-156838-ljhgm"
Oct 30 19:39:40.096760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:40.061663    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z99bw\" (UniqueName: \"kubernetes.io/projected/07ec9d46-74d0-4638-bdbd-67f286deb315-kube-api-access-z99bw\") pod \"maas-api-revoke-keys-e2e-api-b-156838-ljhgm\" (UID: \"07ec9d46-74d0-4638-bdbd-67f286deb315\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-156838-ljhgm"
Oct 30 19:39:40.096760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:40.065458    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:39:40.096760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:40.065632    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:39:40.107006 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:40.106966    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76b4e8e9-54ac-4461-b99c-a6f0361505b6-istio-token\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r\" (UID: \"76b4e8e9-54ac-4461-b99c-a6f0361505b6\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r"
Oct 30 19:39:40.110341 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:40.110314    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64tgh\" (UniqueName: \"kubernetes.io/projected/76b4e8e9-54ac-4461-b99c-a6f0361505b6-kube-api-access-64tgh\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r\" (UID: \"76b4e8e9-54ac-4461-b99c-a6f0361505b6\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r"
Oct 30 19:39:40.111133 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:40.111109    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:39:40.112262 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:40.112239    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76b4e8e9-54ac-4461-b99c-a6f0361505b6-istio-podinfo\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r\" (UID: \"76b4e8e9-54ac-4461-b99c-a6f0361505b6\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r"
Oct 30 19:39:40.216625 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:40.216585    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r"
Oct 30 19:39:40.233638 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:40.231014    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z99bw\" (UniqueName: \"kubernetes.io/projected/07ec9d46-74d0-4638-bdbd-67f286deb315-kube-api-access-z99bw\") pod \"maas-api-revoke-keys-e2e-api-b-156838-ljhgm\" (UID: \"07ec9d46-74d0-4638-bdbd-67f286deb315\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-156838-ljhgm"
Oct 30 19:39:40.233638 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:40.231116    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07ec9d46-74d0-4638-bdbd-67f286deb315-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-156838-ljhgm\" (UID: \"07ec9d46-74d0-4638-bdbd-67f286deb315\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-156838-ljhgm"
Oct 30 19:39:40.233638 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:40.231342    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07ec9d46-74d0-4638-bdbd-67f286deb315-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-156838-ljhgm\" (UID: \"07ec9d46-74d0-4638-bdbd-67f286deb315\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-156838-ljhgm"
Oct 30 19:39:40.233638 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:40.232678    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07ec9d46-74d0-4638-bdbd-67f286deb315-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-156838-ljhgm\" (UID: \"07ec9d46-74d0-4638-bdbd-67f286deb315\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-156838-ljhgm"
Oct 30 19:39:40.247671 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:40.236977    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07ec9d46-74d0-4638-bdbd-67f286deb315-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-156838-ljhgm\" (UID: \"07ec9d46-74d0-4638-bdbd-67f286deb315\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-156838-ljhgm"
Oct 30 19:39:40.247671 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:40.246728    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-1e13df/test-model-1e13df-kserve-c8b994f5f-97xvv"
Oct 30 19:39:40.250584 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:40.250563    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z99bw\" (UniqueName: \"kubernetes.io/projected/07ec9d46-74d0-4638-bdbd-67f286deb315-kube-api-access-z99bw\") pod \"maas-api-revoke-keys-e2e-api-b-156838-ljhgm\" (UID: \"07ec9d46-74d0-4638-bdbd-67f286deb315\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-156838-ljhgm"
Oct 30 19:39:40.335791 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:40.335552    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-1e13df/test-model-1e13df-kserve-c8b994f5f-97xvv"
Oct 30 19:39:40.552881 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:40.552552    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-156838-ljhgm"
Oct 30 19:39:40.998320 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:40.998095    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r"]
Oct 30 19:39:41.044262 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:41.043230    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2d34376-c2wxr"]
Oct 30 19:39:41.044262 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:39:41.044185    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76b4e8e9_54ac_4461_b99c_a6f0361505b6.slice/crio-2b6f80d3e81669eaacc253f7e90c31c1d3d42914e8f39813703931a754646cdc WatchSource:0}: Error finding container 2b6f80d3e81669eaacc253f7e90c31c1d3d42914e8f39813703931a754646cdc: Status 404 returned error can't find the container with id 2b6f80d3e81669eaacc253f7e90c31c1d3d42914e8f39813703931a754646cdc
Oct 30 19:39:41.047443 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:41.047413    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2d34376-c2wxr"]
Oct 30 19:39:41.065854 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:41.065813    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:39:41.066123 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:41.066098    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:39:41.066293 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:41.066273    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:39:41.158656 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:41.158620    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r" event={"ID":"76b4e8e9-54ac-4461-b99c-a6f0361505b6","Type":"ContainerStarted","Data":"2b6f80d3e81669eaacc253f7e90c31c1d3d42914e8f39813703931a754646cdc"}
Oct 30 19:39:41.184881 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:41.179981    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-1e13df/test-model-1e13df-kserve-c8b994f5f-97xvv"
Oct 30 19:39:41.293447 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:41.293352    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-156838-ljhgm"]
Oct 30 19:39:42.178122 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:42.178024    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r" event={"ID":"76b4e8e9-54ac-4461-b99c-a6f0361505b6","Type":"ContainerStarted","Data":"17ea88734869b20fec43ff49cb62bc641cda682522f45e6b64cc406eaeb3daf5"}
Oct 30 19:39:42.181256 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:42.181160    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-156838-ljhgm" event={"ID":"07ec9d46-74d0-4638-bdbd-67f286deb315","Type":"ContainerStarted","Data":"92851e774ff23239be9c38c1d09f854a89149e87eab58b7555226c2bbd0fdbdf"}
Oct 30 19:39:42.181256 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:42.181224    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-156838-ljhgm" event={"ID":"07ec9d46-74d0-4638-bdbd-67f286deb315","Type":"ContainerStarted","Data":"f6f1a0e9f8ea80db0368126fb7d5a48eb23da8b1927aa478750cbccd694563e0"}
Oct 30 19:39:42.203695 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:42.203624    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r" podStartSLOduration=4.203604258 podStartE2EDuration="4.203604258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:42.198357309 +0000 UTC m=+1401.709417275" watchObservedRunningTime="2026-10-30 19:39:42.203604258 +0000 UTC m=+1401.714664221"
Oct 30 19:39:42.220729 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:42.220677    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-156838-ljhgm" podStartSLOduration=3.220661444 podStartE2EDuration="3.220661444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:42.215621606 +0000 UTC m=+1401.726681570" watchObservedRunningTime="2026-10-30 19:39:42.220661444 +0000 UTC m=+1401.731721407"
Oct 30 19:39:42.222999 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:42.222967    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r"
Oct 30 19:39:42.229280 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:42.229252    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r"
Oct 30 19:39:42.481005 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:42.480965    5043 patch_prober.go:29] interesting pod/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:39:42.481215 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:42.481031    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd" podUID="5ff1923b-6547-4e9b-8810-3f79648e3272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:42.655225 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:42.655191    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2db820e0-2c23-42aa-96f4-184da2fe7141" path="/var/lib/kubelet/pods/2db820e0-2c23-42aa-96f4-184da2fe7141/volumes"
Oct 30 19:39:42.827231 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:39:42.827179    5043 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07ec9d46_74d0_4638_bdbd_67f286deb315.slice/crio-conmon-92851e774ff23239be9c38c1d09f854a89149e87eab58b7555226c2bbd0fdbdf.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a040422_5cca_40da_87f7_f5451e7e4b5a.slice/crio-conmon-f2c8ddca63164304aed8bd4e65f71f4b9689d506d776579460dee070bf0e9306.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 19:39:42.914662 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:42.914636    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ae63bf-56c67d7d7b-mgtt5"
Oct 30 19:39:42.927274 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:42.927219    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-d1d256-5b65b66d9d-2djrn" podUID="2f72b377-925e-4664-8b40-1db41ac5fa15" containerName="payload-processing" containerID="cri-o://4f8ab4fb2c02061ec8e432751dfbaa4199ffbc7876a6d87e899c7b6cf3aace03" gracePeriod=25
Oct 30 19:39:43.007897 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.007844    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d1d256-5b889bbcc5-qb7k8" podUID="6fe8aff5-f319-4891-9108-274458478a6e" containerName="payload-pre-processing" containerID="cri-o://df10efb295e320fafbde8f149f56e7a2544b28d181def23037ccb3a49c9bc0cb" gracePeriod=25
Oct 30 19:39:43.083816 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.083100    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a040422-5cca-40da-87f7-f5451e7e4b5a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a040422-5cca-40da-87f7-f5451e7e4b5a-plugins-config-volume\") pod \"9a040422-5cca-40da-87f7-f5451e7e4b5a\" (UID: \"9a040422-5cca-40da-87f7-f5451e7e4b5a\") "
Oct 30 19:39:43.083816 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.083260    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a040422-5cca-40da-87f7-f5451e7e4b5a-kube-api-access-cjn8p\" (UniqueName: \"kubernetes.io/projected/9a040422-5cca-40da-87f7-f5451e7e4b5a-kube-api-access-cjn8p\") pod \"9a040422-5cca-40da-87f7-f5451e7e4b5a\" (UID: \"9a040422-5cca-40da-87f7-f5451e7e4b5a\") "
Oct 30 19:39:43.083816 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.083520    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a040422-5cca-40da-87f7-f5451e7e4b5a-plugins-config-volume" pod "9a040422-5cca-40da-87f7-f5451e7e4b5a" (UID: "9a040422-5cca-40da-87f7-f5451e7e4b5a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:43.086787 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.085163    5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a040422-5cca-40da-87f7-f5451e7e4b5a-plugins-config-volume\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:43.086787 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.086448    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a040422-5cca-40da-87f7-f5451e7e4b5a-kube-api-access-cjn8p" pod "9a040422-5cca-40da-87f7-f5451e7e4b5a" (UID: "9a040422-5cca-40da-87f7-f5451e7e4b5a"). InnerVolumeSpecName "kube-api-access-cjn8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:43.131631 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.131255    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ae63bf-76d99cc4d7-fdtz4"
Oct 30 19:39:43.203130 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.203094    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjn8p\" (UniqueName: \"kubernetes.io/projected/9a040422-5cca-40da-87f7-f5451e7e4b5a-kube-api-access-cjn8p\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:43.220971 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.220934    5043 generic.go:356] "Generic (PLEG): container finished" podID="2f72b377-925e-4664-8b40-1db41ac5fa15" containerID="4f8ab4fb2c02061ec8e432751dfbaa4199ffbc7876a6d87e899c7b6cf3aace03" exitCode=0
Oct 30 19:39:43.221176 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.221008    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d1d256-5b65b66d9d-2djrn" event={"ID":"2f72b377-925e-4664-8b40-1db41ac5fa15","Type":"ContainerDied","Data":"4f8ab4fb2c02061ec8e432751dfbaa4199ffbc7876a6d87e899c7b6cf3aace03"}
Oct 30 19:39:43.223017 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.222990    5043 generic.go:356] "Generic (PLEG): container finished" podID="6fe8aff5-f319-4891-9108-274458478a6e" containerID="df10efb295e320fafbde8f149f56e7a2544b28d181def23037ccb3a49c9bc0cb" exitCode=0
Oct 30 19:39:43.223138 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.223097    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d1d256-5b889bbcc5-qb7k8" event={"ID":"6fe8aff5-f319-4891-9108-274458478a6e","Type":"ContainerDied","Data":"df10efb295e320fafbde8f149f56e7a2544b28d181def23037ccb3a49c9bc0cb"}
Oct 30 19:39:43.224358 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.224336    5043 generic.go:356] "Generic (PLEG): container finished" podID="2c907135-5b39-45a4-b7b5-c54336e1c1a1" containerID="aaeaa38d81cd644c56300b7336c52716102b1524abfd8701c28bc4ef9ad002d1" exitCode=0
Oct 30 19:39:43.224458 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.224408    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ae63bf-76d99cc4d7-fdtz4" event={"ID":"2c907135-5b39-45a4-b7b5-c54336e1c1a1","Type":"ContainerDied","Data":"aaeaa38d81cd644c56300b7336c52716102b1524abfd8701c28bc4ef9ad002d1"}
Oct 30 19:39:43.224458 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.224434    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ae63bf-76d99cc4d7-fdtz4" event={"ID":"2c907135-5b39-45a4-b7b5-c54336e1c1a1","Type":"ContainerDied","Data":"b390897aca1b5c28e70e37d1f66672612547358f01a83952299cffaca2665736"}
Oct 30 19:39:43.224458 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.224454    5043 scope.go:122] "RemoveContainer" containerID="aaeaa38d81cd644c56300b7336c52716102b1524abfd8701c28bc4ef9ad002d1"
Oct 30 19:39:43.224628 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.224574    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ae63bf-76d99cc4d7-fdtz4"
Oct 30 19:39:43.231222 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.227401    5043 generic.go:356] "Generic (PLEG): container finished" podID="07ec9d46-74d0-4638-bdbd-67f286deb315" containerID="92851e774ff23239be9c38c1d09f854a89149e87eab58b7555226c2bbd0fdbdf" exitCode=0
Oct 30 19:39:43.231222 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.227535    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-156838-ljhgm" event={"ID":"07ec9d46-74d0-4638-bdbd-67f286deb315","Type":"ContainerDied","Data":"92851e774ff23239be9c38c1d09f854a89149e87eab58b7555226c2bbd0fdbdf"}
Oct 30 19:39:43.233063 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.232851    5043 generic.go:356] "Generic (PLEG): container finished" podID="9a040422-5cca-40da-87f7-f5451e7e4b5a" containerID="f2c8ddca63164304aed8bd4e65f71f4b9689d506d776579460dee070bf0e9306" exitCode=0
Oct 30 19:39:43.233063 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.232882    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ae63bf-56c67d7d7b-mgtt5" event={"ID":"9a040422-5cca-40da-87f7-f5451e7e4b5a","Type":"ContainerDied","Data":"f2c8ddca63164304aed8bd4e65f71f4b9689d506d776579460dee070bf0e9306"}
Oct 30 19:39:43.233063 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.232921    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ae63bf-56c67d7d7b-mgtt5" event={"ID":"9a040422-5cca-40da-87f7-f5451e7e4b5a","Type":"ContainerDied","Data":"9ad651c2f21214b41acea844a22eb39c3262c2f5ab33bdbae7adcefd2268d728"}
Oct 30 19:39:43.233063 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.232992    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ae63bf-56c67d7d7b-mgtt5"
Oct 30 19:39:43.235047 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.234824    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r"
Oct 30 19:39:43.237099 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.237072    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r"
Oct 30 19:39:43.242423 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.242315    5043 scope.go:122] "RemoveContainer" containerID="aaeaa38d81cd644c56300b7336c52716102b1524abfd8701c28bc4ef9ad002d1"
Oct 30 19:39:43.243002 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:39:43.242860    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaeaa38d81cd644c56300b7336c52716102b1524abfd8701c28bc4ef9ad002d1\": container with ID starting with aaeaa38d81cd644c56300b7336c52716102b1524abfd8701c28bc4ef9ad002d1 not found: ID does not exist" containerID="aaeaa38d81cd644c56300b7336c52716102b1524abfd8701c28bc4ef9ad002d1"
Oct 30 19:39:43.243002 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.242897    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaeaa38d81cd644c56300b7336c52716102b1524abfd8701c28bc4ef9ad002d1"} err="failed to get container status \"aaeaa38d81cd644c56300b7336c52716102b1524abfd8701c28bc4ef9ad002d1\": rpc error: code = NotFound desc = could not find container \"aaeaa38d81cd644c56300b7336c52716102b1524abfd8701c28bc4ef9ad002d1\": container with ID starting with aaeaa38d81cd644c56300b7336c52716102b1524abfd8701c28bc4ef9ad002d1 not found: ID does not exist"
Oct 30 19:39:43.243002 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.242917    5043 scope.go:122] "RemoveContainer" containerID="f2c8ddca63164304aed8bd4e65f71f4b9689d506d776579460dee070bf0e9306"
Oct 30 19:39:43.341795 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.341451    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c907135-5b39-45a4-b7b5-c54336e1c1a1-kube-api-access-9hqg9\" (UniqueName: \"kubernetes.io/projected/2c907135-5b39-45a4-b7b5-c54336e1c1a1-kube-api-access-9hqg9\") pod \"2c907135-5b39-45a4-b7b5-c54336e1c1a1\" (UID: \"2c907135-5b39-45a4-b7b5-c54336e1c1a1\") "
Oct 30 19:39:43.341795 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.341625    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c907135-5b39-45a4-b7b5-c54336e1c1a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c907135-5b39-45a4-b7b5-c54336e1c1a1-plugins-config-volume\") pod \"2c907135-5b39-45a4-b7b5-c54336e1c1a1\" (UID: \"2c907135-5b39-45a4-b7b5-c54336e1c1a1\") "
Oct 30 19:39:43.342166 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.342135    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c907135-5b39-45a4-b7b5-c54336e1c1a1-plugins-config-volume" pod "2c907135-5b39-45a4-b7b5-c54336e1c1a1" (UID: "2c907135-5b39-45a4-b7b5-c54336e1c1a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:43.343426 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.343370    5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c907135-5b39-45a4-b7b5-c54336e1c1a1-plugins-config-volume\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:43.345095 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.345044    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c907135-5b39-45a4-b7b5-c54336e1c1a1-kube-api-access-9hqg9" pod "2c907135-5b39-45a4-b7b5-c54336e1c1a1" (UID: "2c907135-5b39-45a4-b7b5-c54336e1c1a1"). InnerVolumeSpecName "kube-api-access-9hqg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:43.355101 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.355042    5043 scope.go:122] "RemoveContainer" containerID="f2c8ddca63164304aed8bd4e65f71f4b9689d506d776579460dee070bf0e9306"
Oct 30 19:39:43.356887 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:39:43.356850    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2c8ddca63164304aed8bd4e65f71f4b9689d506d776579460dee070bf0e9306\": container with ID starting with f2c8ddca63164304aed8bd4e65f71f4b9689d506d776579460dee070bf0e9306 not found: ID does not exist" containerID="f2c8ddca63164304aed8bd4e65f71f4b9689d506d776579460dee070bf0e9306"
Oct 30 19:39:43.357042 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.356895    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2c8ddca63164304aed8bd4e65f71f4b9689d506d776579460dee070bf0e9306"} err="failed to get container status \"f2c8ddca63164304aed8bd4e65f71f4b9689d506d776579460dee070bf0e9306\": rpc error: code = NotFound desc = could not find container \"f2c8ddca63164304aed8bd4e65f71f4b9689d506d776579460dee070bf0e9306\": container with ID starting with f2c8ddca63164304aed8bd4e65f71f4b9689d506d776579460dee070bf0e9306 not found: ID does not exist"
Oct 30 19:39:43.359790 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.359721    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ae63bf-56c67d7d7b-mgtt5"]
Oct 30 19:39:43.366924 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.366832    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ae63bf-56c67d7d7b-mgtt5"]
Oct 30 19:39:43.421114 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.421079    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d1d256-5b65b66d9d-2djrn"
Oct 30 19:39:43.446794 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.446758    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hqg9\" (UniqueName: \"kubernetes.io/projected/2c907135-5b39-45a4-b7b5-c54336e1c1a1-kube-api-access-9hqg9\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:43.495623 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.495486    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d1d256-5b889bbcc5-qb7k8"
Oct 30 19:39:43.581533 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.580680    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fe8aff5-f319-4891-9108-274458478a6e-kube-api-access-2hxsd\" (UniqueName: \"kubernetes.io/projected/6fe8aff5-f319-4891-9108-274458478a6e-kube-api-access-2hxsd\") pod \"6fe8aff5-f319-4891-9108-274458478a6e\" (UID: \"6fe8aff5-f319-4891-9108-274458478a6e\") "
Oct 30 19:39:43.581533 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.580848    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f72b377-925e-4664-8b40-1db41ac5fa15-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f72b377-925e-4664-8b40-1db41ac5fa15-plugins-config-volume\") pod \"2f72b377-925e-4664-8b40-1db41ac5fa15\" (UID: \"2f72b377-925e-4664-8b40-1db41ac5fa15\") "
Oct 30 19:39:43.588064 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.584552    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fe8aff5-f319-4891-9108-274458478a6e-kube-api-access-2hxsd" pod "6fe8aff5-f319-4891-9108-274458478a6e" (UID: "6fe8aff5-f319-4891-9108-274458478a6e"). InnerVolumeSpecName "kube-api-access-2hxsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:43.588064 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.586625    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f72b377-925e-4664-8b40-1db41ac5fa15-plugins-config-volume" pod "2f72b377-925e-4664-8b40-1db41ac5fa15" (UID: "2f72b377-925e-4664-8b40-1db41ac5fa15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:43.589759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.588931    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fe8aff5-f319-4891-9108-274458478a6e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fe8aff5-f319-4891-9108-274458478a6e-plugins-config-volume\") pod \"6fe8aff5-f319-4891-9108-274458478a6e\" (UID: \"6fe8aff5-f319-4891-9108-274458478a6e\") "
Oct 30 19:39:43.589759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.589151    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f72b377-925e-4664-8b40-1db41ac5fa15-kube-api-access-f6m8t\" (UniqueName: \"kubernetes.io/projected/2f72b377-925e-4664-8b40-1db41ac5fa15-kube-api-access-f6m8t\") pod \"2f72b377-925e-4664-8b40-1db41ac5fa15\" (UID: \"2f72b377-925e-4664-8b40-1db41ac5fa15\") "
Oct 30 19:39:43.592497 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.592128    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hxsd\" (UniqueName: \"kubernetes.io/projected/6fe8aff5-f319-4891-9108-274458478a6e-kube-api-access-2hxsd\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:43.592497 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.592159    5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f72b377-925e-4664-8b40-1db41ac5fa15-plugins-config-volume\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:43.597787 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.596238    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f72b377-925e-4664-8b40-1db41ac5fa15-kube-api-access-f6m8t" pod "2f72b377-925e-4664-8b40-1db41ac5fa15" (UID: "2f72b377-925e-4664-8b40-1db41ac5fa15"). InnerVolumeSpecName "kube-api-access-f6m8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:43.614903 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.613122    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fe8aff5-f319-4891-9108-274458478a6e-plugins-config-volume" pod "6fe8aff5-f319-4891-9108-274458478a6e" (UID: "6fe8aff5-f319-4891-9108-274458478a6e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:43.666393 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.666310    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ae63bf-76d99cc4d7-fdtz4"]
Oct 30 19:39:43.678009 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.675535    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ae63bf-76d99cc4d7-fdtz4"]
Oct 30 19:39:43.694045 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.694001    5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fe8aff5-f319-4891-9108-274458478a6e-plugins-config-volume\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:43.694045 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:43.694042    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6m8t\" (UniqueName: \"kubernetes.io/projected/2f72b377-925e-4664-8b40-1db41ac5fa15-kube-api-access-f6m8t\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:44.243463 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:44.243428    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d1d256-5b65b66d9d-2djrn"
Oct 30 19:39:44.243463 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:44.243445    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d1d256-5b65b66d9d-2djrn" event={"ID":"2f72b377-925e-4664-8b40-1db41ac5fa15","Type":"ContainerDied","Data":"99b62b9ebbcf88ff12add130e083fb03a1af25a13908baf21e951d69f4d62548"}
Oct 30 19:39:44.244032 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:44.243505    5043 scope.go:122] "RemoveContainer" containerID="4f8ab4fb2c02061ec8e432751dfbaa4199ffbc7876a6d87e899c7b6cf3aace03"
Oct 30 19:39:44.252528 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:44.245397    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d1d256-5b889bbcc5-qb7k8"
Oct 30 19:39:44.252528 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:44.245409    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d1d256-5b889bbcc5-qb7k8" event={"ID":"6fe8aff5-f319-4891-9108-274458478a6e","Type":"ContainerDied","Data":"61b17590a7c535476b39208e70efcd26d85eb7bfe4ca5b3326588f3a7b59a14a"}
Oct 30 19:39:44.257879 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:44.257854    5043 scope.go:122] "RemoveContainer" containerID="df10efb295e320fafbde8f149f56e7a2544b28d181def23037ccb3a49c9bc0cb"
Oct 30 19:39:44.273154 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:44.273123    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d1d256-5b65b66d9d-2djrn"]
Oct 30 19:39:44.278258 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:44.278233    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d1d256-5b65b66d9d-2djrn"]
Oct 30 19:39:44.291959 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:44.291911    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d1d256-5b889bbcc5-qb7k8"]
Oct 30 19:39:44.303309 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:44.303275    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d1d256-5b889bbcc5-qb7k8"]
Oct 30 19:39:44.550863 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:44.550778    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-156838-ljhgm"
Oct 30 19:39:44.614894 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:44.613659    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07ec9d46-74d0-4638-bdbd-67f286deb315-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07ec9d46-74d0-4638-bdbd-67f286deb315-maas-api-xks-ca\") pod \"07ec9d46-74d0-4638-bdbd-67f286deb315\" (UID: \"07ec9d46-74d0-4638-bdbd-67f286deb315\") "
Oct 30 19:39:44.614894 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:44.613864    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07ec9d46-74d0-4638-bdbd-67f286deb315-kube-api-access-z99bw\" (UniqueName: \"kubernetes.io/projected/07ec9d46-74d0-4638-bdbd-67f286deb315-kube-api-access-z99bw\") pod \"07ec9d46-74d0-4638-bdbd-67f286deb315\" (UID: \"07ec9d46-74d0-4638-bdbd-67f286deb315\") "
Oct 30 19:39:44.614894 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:44.613920    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07ec9d46-74d0-4638-bdbd-67f286deb315-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07ec9d46-74d0-4638-bdbd-67f286deb315-maas-api-openshift-ca\") pod \"07ec9d46-74d0-4638-bdbd-67f286deb315\" (UID: \"07ec9d46-74d0-4638-bdbd-67f286deb315\") "
Oct 30 19:39:44.617214 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:44.617155    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07ec9d46-74d0-4638-bdbd-67f286deb315-maas-api-openshift-ca" pod "07ec9d46-74d0-4638-bdbd-67f286deb315" (UID: "07ec9d46-74d0-4638-bdbd-67f286deb315"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:44.619539 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:44.619513    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07ec9d46-74d0-4638-bdbd-67f286deb315-maas-api-xks-ca" pod "07ec9d46-74d0-4638-bdbd-67f286deb315" (UID: "07ec9d46-74d0-4638-bdbd-67f286deb315"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:39:44.620328 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:44.620302    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07ec9d46-74d0-4638-bdbd-67f286deb315-kube-api-access-z99bw" pod "07ec9d46-74d0-4638-bdbd-67f286deb315" (UID: "07ec9d46-74d0-4638-bdbd-67f286deb315"). InnerVolumeSpecName "kube-api-access-z99bw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:44.652716 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:44.652687    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c907135-5b39-45a4-b7b5-c54336e1c1a1" path="/var/lib/kubelet/pods/2c907135-5b39-45a4-b7b5-c54336e1c1a1/volumes"
Oct 30 19:39:44.653226 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:44.653210    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f72b377-925e-4664-8b40-1db41ac5fa15" path="/var/lib/kubelet/pods/2f72b377-925e-4664-8b40-1db41ac5fa15/volumes"
Oct 30 19:39:44.653621 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:44.653607    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fe8aff5-f319-4891-9108-274458478a6e" path="/var/lib/kubelet/pods/6fe8aff5-f319-4891-9108-274458478a6e/volumes"
Oct 30 19:39:44.654003 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:44.653990    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a040422-5cca-40da-87f7-f5451e7e4b5a" path="/var/lib/kubelet/pods/9a040422-5cca-40da-87f7-f5451e7e4b5a/volumes"
Oct 30 19:39:44.716598 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:44.716565    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07ec9d46-74d0-4638-bdbd-67f286deb315-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:44.716598 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:44.716599    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z99bw\" (UniqueName: \"kubernetes.io/projected/07ec9d46-74d0-4638-bdbd-67f286deb315-kube-api-access-z99bw\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:44.716925 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:44.716612    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07ec9d46-74d0-4638-bdbd-67f286deb315-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:45.275215 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:45.275177    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-156838-ljhgm"
Oct 30 19:39:45.275821 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:45.275321    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-156838-ljhgm" event={"ID":"07ec9d46-74d0-4638-bdbd-67f286deb315","Type":"ContainerDied","Data":"f6f1a0e9f8ea80db0368126fb7d5a48eb23da8b1927aa478750cbccd694563e0"}
Oct 30 19:39:45.275821 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:45.275364    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6f1a0e9f8ea80db0368126fb7d5a48eb23da8b1927aa478750cbccd694563e0"
Oct 30 19:39:45.548996 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:45.548910    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ae63bf-5xlsr"]
Oct 30 19:39:45.555078 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:45.554872    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ae63bf-5xlsr"]
Oct 30 19:39:46.208223 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:46.208180    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ac37c8-57cb47d8c-4cprd"]
Oct 30 19:39:46.209042 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:46.209020    5043 cpu_manager.go:414] "containerMap: removing container" podUID="2f72b377-925e-4664-8b40-1db41ac5fa15" containerName="payload-processing"
Oct 30 19:39:46.209171 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:46.209044    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f72b377-925e-4664-8b40-1db41ac5fa15" containerName="payload-processing"
Oct 30 19:39:46.209171 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:46.209080    5043 cpu_manager.go:414] "containerMap: removing container" podUID="9a040422-5cca-40da-87f7-f5451e7e4b5a" containerName="payload-processing"
Oct 30 19:39:46.209171 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:46.209088    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a040422-5cca-40da-87f7-f5451e7e4b5a" containerName="payload-processing"
Oct 30 19:39:46.209171 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:46.209107    5043 cpu_manager.go:414] "containerMap: removing container" podUID="07ec9d46-74d0-4638-bdbd-67f286deb315" containerName="revoke-keys"
Oct 30 19:39:46.209171 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:46.209114    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07ec9d46-74d0-4638-bdbd-67f286deb315" containerName="revoke-keys"
Oct 30 19:39:46.209171 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:46.209138    5043 cpu_manager.go:414] "containerMap: removing container" podUID="2c907135-5b39-45a4-b7b5-c54336e1c1a1" containerName="payload-pre-processing"
Oct 30 19:39:46.209171 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:46.209145    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c907135-5b39-45a4-b7b5-c54336e1c1a1" containerName="payload-pre-processing"
Oct 30 19:39:46.209532 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:46.209181    5043 cpu_manager.go:414] "containerMap: removing container" podUID="6fe8aff5-f319-4891-9108-274458478a6e" containerName="payload-pre-processing"
Oct 30 19:39:46.209532 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:46.209188    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fe8aff5-f319-4891-9108-274458478a6e" containerName="payload-pre-processing"
Oct 30 19:39:46.209532 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:46.209360    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fe8aff5-f319-4891-9108-274458478a6e" containerName="payload-pre-processing"
Oct 30 19:39:46.209532 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:46.209388    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f72b377-925e-4664-8b40-1db41ac5fa15" containerName="payload-processing"
Oct 30 19:39:46.209532 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:46.209405    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a040422-5cca-40da-87f7-f5451e7e4b5a" containerName="payload-processing"
Oct 30 19:39:46.209532 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:46.209414    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="07ec9d46-74d0-4638-bdbd-67f286deb315" containerName="revoke-keys"
Oct 30 19:39:46.209532 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:46.209427    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c907135-5b39-45a4-b7b5-c54336e1c1a1" containerName="payload-pre-processing"
Oct 30 19:39:46.239086 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:46.234295    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ac37c8-69dbd9fb6d-xjzdf"]
Oct 30 19:39:46.239086 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:46.234428    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ac37c8-57cb47d8c-4cprd"
Oct 30 19:39:46.242898 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:46.242872    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-ac37c8\""
Oct 30 19:39:46.268175 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:46.268133    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ac37c8-57cb47d8c-4cprd"]
Oct 30 19:39:46.268367 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:46.268206    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ac37c8-69dbd9fb6d-xjzdf"]
Oct 30 19:39:46.268367 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:46.268332    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ac37c8-69dbd9fb6d-xjzdf"
Oct 30 19:39:46.273813 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:46.273782    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-ac37c8-dockercfg-s8pcg\""
Oct 30 19:39:46.297987 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:46.297944    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42cfcd85-0bda-4e33-ae8c-f84eb90afd6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ac37c8-57cb47d8c-4cprd\" (UID: \"42cfcd85-0bda-4e33-ae8c-f84eb90afd6b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ac37c8-57cb47d8c-4cprd"
Oct 30 19:39:46.298496 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:46.298084    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfx66\" (UniqueName: \"kubernetes.io/projected/42cfcd85-0bda-4e33-ae8c-f84eb90afd6b-kube-api-access-nfx66\") pod \"payload-pre-processing-e2e-inf-b-ac37c8-57cb47d8c-4cprd\" (UID: \"42cfcd85-0bda-4e33-ae8c-f84eb90afd6b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ac37c8-57cb47d8c-4cprd"
Oct 30 19:39:46.409894 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:46.407909    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwx8b\" (UniqueName: \"kubernetes.io/projected/1ed7ff0c-6ec2-443f-b1d8-8f7cbf8238ce-kube-api-access-cwx8b\") pod \"payload-processing-e2e-inf-b-ac37c8-69dbd9fb6d-xjzdf\" (UID: \"1ed7ff0c-6ec2-443f-b1d8-8f7cbf8238ce\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ac37c8-69dbd9fb6d-xjzdf"
Oct 30 19:39:46.409894 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:46.407984    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfx66\" (UniqueName: \"kubernetes.io/projected/42cfcd85-0bda-4e33-ae8c-f84eb90afd6b-kube-api-access-nfx66\") pod \"payload-pre-processing-e2e-inf-b-ac37c8-57cb47d8c-4cprd\" (UID: \"42cfcd85-0bda-4e33-ae8c-f84eb90afd6b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ac37c8-57cb47d8c-4cprd"
Oct 30 19:39:46.409894 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:46.408059    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ed7ff0c-6ec2-443f-b1d8-8f7cbf8238ce-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ac37c8-69dbd9fb6d-xjzdf\" (UID: \"1ed7ff0c-6ec2-443f-b1d8-8f7cbf8238ce\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ac37c8-69dbd9fb6d-xjzdf"
Oct 30 19:39:46.409894 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:46.408345    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42cfcd85-0bda-4e33-ae8c-f84eb90afd6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ac37c8-57cb47d8c-4cprd\" (UID: \"42cfcd85-0bda-4e33-ae8c-f84eb90afd6b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ac37c8-57cb47d8c-4cprd"
Oct 30 19:39:46.409894 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:46.409445    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42cfcd85-0bda-4e33-ae8c-f84eb90afd6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ac37c8-57cb47d8c-4cprd\" (UID: \"42cfcd85-0bda-4e33-ae8c-f84eb90afd6b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ac37c8-57cb47d8c-4cprd"
Oct 30 19:39:46.414245 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:46.413875    5043 patch_prober.go:29] interesting pod/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:39:46.414245 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:46.413941    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx" podUID="f791cec1-a1e4-4067-9221-cc04dafba0eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:39:46.486677 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:46.486539    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfx66\" (UniqueName: \"kubernetes.io/projected/42cfcd85-0bda-4e33-ae8c-f84eb90afd6b-kube-api-access-nfx66\") pod \"payload-pre-processing-e2e-inf-b-ac37c8-57cb47d8c-4cprd\" (UID: \"42cfcd85-0bda-4e33-ae8c-f84eb90afd6b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ac37c8-57cb47d8c-4cprd"
Oct 30 19:39:46.516289 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:46.511326    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwx8b\" (UniqueName: \"kubernetes.io/projected/1ed7ff0c-6ec2-443f-b1d8-8f7cbf8238ce-kube-api-access-cwx8b\") pod \"payload-processing-e2e-inf-b-ac37c8-69dbd9fb6d-xjzdf\" (UID: \"1ed7ff0c-6ec2-443f-b1d8-8f7cbf8238ce\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ac37c8-69dbd9fb6d-xjzdf"
Oct 30 19:39:46.516289 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:46.511446    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ed7ff0c-6ec2-443f-b1d8-8f7cbf8238ce-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ac37c8-69dbd9fb6d-xjzdf\" (UID: \"1ed7ff0c-6ec2-443f-b1d8-8f7cbf8238ce\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ac37c8-69dbd9fb6d-xjzdf"
Oct 30 19:39:46.532759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:46.529485    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ed7ff0c-6ec2-443f-b1d8-8f7cbf8238ce-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ac37c8-69dbd9fb6d-xjzdf\" (UID: \"1ed7ff0c-6ec2-443f-b1d8-8f7cbf8238ce\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ac37c8-69dbd9fb6d-xjzdf"
Oct 30 19:39:46.540470 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:46.540391    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwx8b\" (UniqueName: \"kubernetes.io/projected/1ed7ff0c-6ec2-443f-b1d8-8f7cbf8238ce-kube-api-access-cwx8b\") pod \"payload-processing-e2e-inf-b-ac37c8-69dbd9fb6d-xjzdf\" (UID: \"1ed7ff0c-6ec2-443f-b1d8-8f7cbf8238ce\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ac37c8-69dbd9fb6d-xjzdf"
Oct 30 19:39:46.583371 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:46.583100    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ac37c8-57cb47d8c-4cprd"
Oct 30 19:39:46.608756 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:46.606063    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ac37c8-69dbd9fb6d-xjzdf"
Oct 30 19:39:46.653479 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:46.653446    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39cf0453-c5af-4fba-8a9c-5d3ace5dc747" path="/var/lib/kubelet/pods/39cf0453-c5af-4fba-8a9c-5d3ace5dc747/volumes"
Oct 30 19:39:47.338849 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:46.983665    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ac37c8-57cb47d8c-4cprd"]
Oct 30 19:39:47.338849 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:39:46.986295    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42cfcd85_0bda_4e33_ae8c_f84eb90afd6b.slice/crio-2b4c521f80eb333941f899463e85f28336df8cbff7267130d349dd545d46c928 WatchSource:0}: Error finding container 2b4c521f80eb333941f899463e85f28336df8cbff7267130d349dd545d46c928: Status 404 returned error can't find the container with id 2b4c521f80eb333941f899463e85f28336df8cbff7267130d349dd545d46c928
Oct 30 19:39:47.338849 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:47.003837    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ac37c8-69dbd9fb6d-xjzdf"]
Oct 30 19:39:47.338849 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:47.309941    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ac37c8-57cb47d8c-4cprd" event={"ID":"42cfcd85-0bda-4e33-ae8c-f84eb90afd6b","Type":"ContainerStarted","Data":"a85ae4b85ef525013d42625b7cbae6c62a0d89dfa40311c97d4094706e64536e"}
Oct 30 19:39:47.338849 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:47.310010    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ac37c8-57cb47d8c-4cprd"
Oct 30 19:39:47.338849 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:47.310034    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ac37c8-57cb47d8c-4cprd" event={"ID":"42cfcd85-0bda-4e33-ae8c-f84eb90afd6b","Type":"ContainerStarted","Data":"2b4c521f80eb333941f899463e85f28336df8cbff7267130d349dd545d46c928"}
Oct 30 19:39:47.338849 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:47.311685    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ac37c8-69dbd9fb6d-xjzdf" event={"ID":"1ed7ff0c-6ec2-443f-b1d8-8f7cbf8238ce","Type":"ContainerStarted","Data":"e2b6fc6754c1988efc99e2385cb6d20b4acd29a3db0a2809adaf7febf11279be"}
Oct 30 19:39:47.338849 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:47.311719    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ac37c8-69dbd9fb6d-xjzdf" event={"ID":"1ed7ff0c-6ec2-443f-b1d8-8f7cbf8238ce","Type":"ContainerStarted","Data":"276d32cce14813972e3670ee2a0a093dd88c31ec00f80ba41e619a15628a4c9e"}
Oct 30 19:39:47.338849 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:47.311827    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ac37c8-69dbd9fb6d-xjzdf"
Oct 30 19:39:47.342791 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:47.342731    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ac37c8-57cb47d8c-4cprd" podStartSLOduration=1.342718067 podStartE2EDuration="1.342718067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:47.338464926 +0000 UTC m=+1406.849524895" watchObservedRunningTime="2026-10-30 19:39:47.342718067 +0000 UTC m=+1406.853778082"
Oct 30 19:39:47.372680 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:47.372612    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-ac37c8-69dbd9fb6d-xjzdf" podStartSLOduration=1.372592779 podStartE2EDuration="1.372592779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:47.367292796 +0000 UTC m=+1406.878352759" watchObservedRunningTime="2026-10-30 19:39:47.372592779 +0000 UTC m=+1406.883652744"
Oct 30 19:39:48.324189 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:48.324158    5043 generic.go:356] "Generic (PLEG): container finished" podID="5ff1923b-6547-4e9b-8810-3f79648e3272" containerID="7ec243dad6c669aa3939aa3a8751708fa5f41444c4da0a374873dc8b9e1e3f16" exitCode=0
Oct 30 19:39:48.324419 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:48.324233    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd" event={"ID":"5ff1923b-6547-4e9b-8810-3f79648e3272","Type":"ContainerDied","Data":"7ec243dad6c669aa3939aa3a8751708fa5f41444c4da0a374873dc8b9e1e3f16"}
Oct 30 19:39:48.324729 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:48.324710    5043 scope.go:122] "RemoveContainer" containerID="7ec243dad6c669aa3939aa3a8751708fa5f41444c4da0a374873dc8b9e1e3f16"
Oct 30 19:39:48.326163 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:48.326144    5043 generic.go:356] "Generic (PLEG): container finished" podID="f791cec1-a1e4-4067-9221-cc04dafba0eb" containerID="2bf0d118cce4ea36dc19bc4338eaf9c3cf4336991c115f7aa305eedaa4b76956" exitCode=0
Oct 30 19:39:48.326266 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:48.326216    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx" event={"ID":"f791cec1-a1e4-4067-9221-cc04dafba0eb","Type":"ContainerDied","Data":"2bf0d118cce4ea36dc19bc4338eaf9c3cf4336991c115f7aa305eedaa4b76956"}
Oct 30 19:39:48.326780 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:48.326751    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:39:48.326882 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:48.326808    5043 scope.go:122] "RemoveContainer" containerID="2bf0d118cce4ea36dc19bc4338eaf9c3cf4336991c115f7aa305eedaa4b76956"
Oct 30 19:39:48.326882 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:48.326836    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:39:48.326959 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:48.326883    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:39:48.328443 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:48.328418    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:39:48.328519 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:48.328481    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:39:48.328519 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:48.328516    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:39:49.338486 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:49.338454    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx" event={"ID":"f791cec1-a1e4-4067-9221-cc04dafba0eb","Type":"ContainerStarted","Data":"cb07df21823bf0822cc28be90ffa63cfdc67e892be4a904cc079f6e990725195"}
Oct 30 19:39:49.340711 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:49.340688    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd" event={"ID":"5ff1923b-6547-4e9b-8810-3f79648e3272","Type":"ContainerStarted","Data":"d4061dda458306dde91ff8f2a254ce9bb14709f6ea844c758630cff33d1aa8a4"}
Oct 30 19:39:49.490761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:49.482210    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx"
Oct 30 19:39:49.490761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:49.482345    5043 patch_prober.go:29] interesting pod/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 30 19:39:49.490761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:49.482390    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx" podUID="f791cec1-a1e4-4067-9221-cc04dafba0eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 30 19:39:49.583785 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:49.569297    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd"
Oct 30 19:39:49.583785 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:49.569439    5043 patch_prober.go:29] interesting pod/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Oct 30 19:39:49.583785 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:49.569485    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd" podUID="5ff1923b-6547-4e9b-8810-3f79648e3272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Oct 30 19:39:50.052712 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:50.052566    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-156838-79f475dcd-x2qg9"]
Oct 30 19:39:50.059721 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:50.059688    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-156838-6749758ddc-tnv4d"]
Oct 30 19:39:50.480810 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:50.472251    5043 patch_prober.go:29] interesting pod/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 30 19:39:50.480810 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:50.472309    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx" podUID="f791cec1-a1e4-4067-9221-cc04dafba0eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 30 19:39:50.568826 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:50.568174    5043 patch_prober.go:29] interesting pod/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Oct 30 19:39:50.568826 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:50.568230    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd" podUID="5ff1923b-6547-4e9b-8810-3f79648e3272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Oct 30 19:39:51.472762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:51.472187    5043 patch_prober.go:29] interesting pod/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 30 19:39:51.472762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:51.472256    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx" podUID="f791cec1-a1e4-4067-9221-cc04dafba0eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 30 19:39:51.566173 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:51.566135    5043 patch_prober.go:29] interesting pod/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Oct 30 19:39:51.566819 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:51.566788    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd" podUID="5ff1923b-6547-4e9b-8810-3f79648e3272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Oct 30 19:39:52.477763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:52.472180    5043 patch_prober.go:29] interesting pod/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 30 19:39:52.477763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:52.472248    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx" podUID="f791cec1-a1e4-4067-9221-cc04dafba0eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 30 19:39:52.565929 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:52.565879    5043 patch_prober.go:29] interesting pod/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Oct 30 19:39:52.566141 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:52.565946    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd" podUID="5ff1923b-6547-4e9b-8810-3f79648e3272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Oct 30 19:39:53.479764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:53.479139    5043 patch_prober.go:29] interesting pod/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 30 19:39:53.479764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:53.479206    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx" podUID="f791cec1-a1e4-4067-9221-cc04dafba0eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 30 19:39:53.571763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:53.567136    5043 patch_prober.go:29] interesting pod/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Oct 30 19:39:53.571763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:53.567201    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd" podUID="5ff1923b-6547-4e9b-8810-3f79648e3272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Oct 30 19:39:54.483761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:54.480019    5043 patch_prober.go:29] interesting pod/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 30 19:39:54.483761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:54.480094    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx" podUID="f791cec1-a1e4-4067-9221-cc04dafba0eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 30 19:39:54.595775 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:54.594932    5043 patch_prober.go:29] interesting pod/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Oct 30 19:39:54.595775 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:54.594996    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd" podUID="5ff1923b-6547-4e9b-8810-3f79648e3272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Oct 30 19:39:55.053053 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:55.052927    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-156838-79f475dcd-x2qg9" podUID="9db7ea84-934d-41d6-aa27-c163dfc862c5" containerName="payload-processing" containerID="cri-o://0581936c34a1300d468eeadbd7c5aa7cfe092626c6db1d6c943d30283cd7f9cb" gracePeriod=25
Oct 30 19:39:55.087840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:55.087790    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-156838-6749758ddc-tnv4d" podUID="21041d00-c41b-4b80-839f-c48c0f6988c7" containerName="payload-pre-processing" containerID="cri-o://21054e68e815c5ae4fd338a67895e5d5f3c58e0ef4d5c977521c9421d176a553" gracePeriod=25
Oct 30 19:39:55.427765 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:39:55.411953    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:39:55.496770 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:55.485862    5043 patch_prober.go:29] interesting pod/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body=
Oct 30 19:39:55.496770 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:55.485933    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx" podUID="f791cec1-a1e4-4067-9221-cc04dafba0eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused"
Oct 30 19:39:55.650700 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:55.567335    5043 patch_prober.go:29] interesting pod/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Oct 30 19:39:55.650700 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:55.567411    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd" podUID="5ff1923b-6547-4e9b-8810-3f79648e3272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Oct 30 19:39:55.650700 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:55.598408    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t"]
Oct 30 19:39:55.650700 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:55.598634    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t" podUID="88d33c34-9387-4234-b953-446167850092" containerName="istio-proxy" containerID="cri-o://53a012b0030f36e1b61ed75f81affb415ca4eaa1f4182837fd73d5322d6c7752" gracePeriod=30
Oct 30 19:39:55.709762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:55.690312    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t"]
Oct 30 19:39:55.709762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:55.690588    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t" podUID="4897f8f5-579f-45ed-a964-f2379eb52848" containerName="istio-proxy" containerID="cri-o://ed9f07518d42efa6ff32cfd0e47b68eded8fba60afb242e22f90fcffff274993" gracePeriod=30
Oct 30 19:39:55.899770 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:55.899602    5043 generic.go:356] "Generic (PLEG): container finished" podID="9db7ea84-934d-41d6-aa27-c163dfc862c5" containerID="0581936c34a1300d468eeadbd7c5aa7cfe092626c6db1d6c943d30283cd7f9cb" exitCode=0
Oct 30 19:39:55.900433 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:55.900212    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-156838-79f475dcd-x2qg9" event={"ID":"9db7ea84-934d-41d6-aa27-c163dfc862c5","Type":"ContainerDied","Data":"0581936c34a1300d468eeadbd7c5aa7cfe092626c6db1d6c943d30283cd7f9cb"}
Oct 30 19:39:55.904170 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:55.904097    5043 generic.go:356] "Generic (PLEG): container finished" podID="21041d00-c41b-4b80-839f-c48c0f6988c7" containerID="21054e68e815c5ae4fd338a67895e5d5f3c58e0ef4d5c977521c9421d176a553" exitCode=0
Oct 30 19:39:55.904486 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:55.904463    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-156838-6749758ddc-tnv4d" event={"ID":"21041d00-c41b-4b80-839f-c48c0f6988c7","Type":"ContainerDied","Data":"21054e68e815c5ae4fd338a67895e5d5f3c58e0ef4d5c977521c9421d176a553"}
Oct 30 19:39:56.113372 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.113327    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-156838-6749758ddc-tnv4d"
Oct 30 19:39:56.213758 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.213151    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21041d00-c41b-4b80-839f-c48c0f6988c7-kube-api-access-m9hmf\" (UniqueName: \"kubernetes.io/projected/21041d00-c41b-4b80-839f-c48c0f6988c7-kube-api-access-m9hmf\") pod \"21041d00-c41b-4b80-839f-c48c0f6988c7\" (UID: \"21041d00-c41b-4b80-839f-c48c0f6988c7\") "
Oct 30 19:39:56.216159 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.214286    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21041d00-c41b-4b80-839f-c48c0f6988c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21041d00-c41b-4b80-839f-c48c0f6988c7-plugins-config-volume\") pod \"21041d00-c41b-4b80-839f-c48c0f6988c7\" (UID: \"21041d00-c41b-4b80-839f-c48c0f6988c7\") "
Oct 30 19:39:56.216159 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.215521    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21041d00-c41b-4b80-839f-c48c0f6988c7-plugins-config-volume" pod "21041d00-c41b-4b80-839f-c48c0f6988c7" (UID: "21041d00-c41b-4b80-839f-c48c0f6988c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:56.267245 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.250422    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21041d00-c41b-4b80-839f-c48c0f6988c7-kube-api-access-m9hmf" pod "21041d00-c41b-4b80-839f-c48c0f6988c7" (UID: "21041d00-c41b-4b80-839f-c48c0f6988c7"). InnerVolumeSpecName "kube-api-access-m9hmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:56.300778 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.300704    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-156838-79f475dcd-x2qg9"
Oct 30 19:39:56.316390 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.316336    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9db7ea84-934d-41d6-aa27-c163dfc862c5-kube-api-access-tjvrm\" (UniqueName: \"kubernetes.io/projected/9db7ea84-934d-41d6-aa27-c163dfc862c5-kube-api-access-tjvrm\") pod \"9db7ea84-934d-41d6-aa27-c163dfc862c5\" (UID: \"9db7ea84-934d-41d6-aa27-c163dfc862c5\") "
Oct 30 19:39:56.316583 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.316503    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9db7ea84-934d-41d6-aa27-c163dfc862c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9db7ea84-934d-41d6-aa27-c163dfc862c5-plugins-config-volume\") pod \"9db7ea84-934d-41d6-aa27-c163dfc862c5\" (UID: \"9db7ea84-934d-41d6-aa27-c163dfc862c5\") "
Oct 30 19:39:56.317300 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.317256    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9db7ea84-934d-41d6-aa27-c163dfc862c5-plugins-config-volume" pod "9db7ea84-934d-41d6-aa27-c163dfc862c5" (UID: "9db7ea84-934d-41d6-aa27-c163dfc862c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:56.318008 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.317767    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9hmf\" (UniqueName: \"kubernetes.io/projected/21041d00-c41b-4b80-839f-c48c0f6988c7-kube-api-access-m9hmf\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:56.318008 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.317802    5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9db7ea84-934d-41d6-aa27-c163dfc862c5-plugins-config-volume\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:56.318008 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.317821    5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21041d00-c41b-4b80-839f-c48c0f6988c7-plugins-config-volume\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:56.320782 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.319840    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9db7ea84-934d-41d6-aa27-c163dfc862c5-kube-api-access-tjvrm" pod "9db7ea84-934d-41d6-aa27-c163dfc862c5" (UID: "9db7ea84-934d-41d6-aa27-c163dfc862c5"). InnerVolumeSpecName "kube-api-access-tjvrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:56.419135 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.419094    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjvrm\" (UniqueName: \"kubernetes.io/projected/9db7ea84-934d-41d6-aa27-c163dfc862c5-kube-api-access-tjvrm\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:56.440485 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.440447    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd"
Oct 30 19:39:56.475982 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.475907    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx"
Oct 30 19:39:56.476368 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.476348    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx"
Oct 30 19:39:56.477162 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.477141    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx"
Oct 30 19:39:56.571237 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.571208    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd"
Oct 30 19:39:56.596535 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.596506    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj"]
Oct 30 19:39:56.597483 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.597461    5043 cpu_manager.go:414] "containerMap: removing container" podUID="9db7ea84-934d-41d6-aa27-c163dfc862c5" containerName="payload-processing"
Oct 30 19:39:56.597483 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.597481    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9db7ea84-934d-41d6-aa27-c163dfc862c5" containerName="payload-processing"
Oct 30 19:39:56.597635 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.597512    5043 cpu_manager.go:414] "containerMap: removing container" podUID="21041d00-c41b-4b80-839f-c48c0f6988c7" containerName="payload-pre-processing"
Oct 30 19:39:56.597635 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.597517    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21041d00-c41b-4b80-839f-c48c0f6988c7" containerName="payload-pre-processing"
Oct 30 19:39:56.597707 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.597643    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="9db7ea84-934d-41d6-aa27-c163dfc862c5" containerName="payload-processing"
Oct 30 19:39:56.597707 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.597658    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="21041d00-c41b-4b80-839f-c48c0f6988c7" containerName="payload-pre-processing"
Oct 30 19:39:56.601483 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.601466    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj"
Oct 30 19:39:56.607856 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.607833    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-89a3f6-openshift-default-dockercfg-kjltj\""
Oct 30 19:39:56.617171 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.617141    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj"]
Oct 30 19:39:56.622727 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.622699    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/097a37e6-fa47-4514-b43d-5c462533947b-istio-token\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj\" (UID: \"097a37e6-fa47-4514-b43d-5c462533947b\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj"
Oct 30 19:39:56.622905 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.622816    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/097a37e6-fa47-4514-b43d-5c462533947b-workload-certs\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj\" (UID: \"097a37e6-fa47-4514-b43d-5c462533947b\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj"
Oct 30 19:39:56.622973 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.622945    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/097a37e6-fa47-4514-b43d-5c462533947b-istiod-ca-cert\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj\" (UID: \"097a37e6-fa47-4514-b43d-5c462533947b\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj"
Oct 30 19:39:56.623092 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.623069    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/097a37e6-fa47-4514-b43d-5c462533947b-workload-socket\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj\" (UID: \"097a37e6-fa47-4514-b43d-5c462533947b\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj"
Oct 30 19:39:56.623220 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.623112    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/097a37e6-fa47-4514-b43d-5c462533947b-istio-envoy\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj\" (UID: \"097a37e6-fa47-4514-b43d-5c462533947b\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj"
Oct 30 19:39:56.623304 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.623218    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/097a37e6-fa47-4514-b43d-5c462533947b-istio-data\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj\" (UID: \"097a37e6-fa47-4514-b43d-5c462533947b\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj"
Oct 30 19:39:56.623304 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.623263    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/097a37e6-fa47-4514-b43d-5c462533947b-credential-socket\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj\" (UID: \"097a37e6-fa47-4514-b43d-5c462533947b\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj"
Oct 30 19:39:56.623455 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.623316    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/097a37e6-fa47-4514-b43d-5c462533947b-istio-podinfo\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj\" (UID: \"097a37e6-fa47-4514-b43d-5c462533947b\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj"
Oct 30 19:39:56.623455 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.623425    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9mfw\" (UniqueName: \"kubernetes.io/projected/097a37e6-fa47-4514-b43d-5c462533947b-kube-api-access-l9mfw\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj\" (UID: \"097a37e6-fa47-4514-b43d-5c462533947b\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj"
Oct 30 19:39:56.724932 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.724894    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/097a37e6-fa47-4514-b43d-5c462533947b-workload-certs\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj\" (UID: \"097a37e6-fa47-4514-b43d-5c462533947b\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj"
Oct 30 19:39:56.725100 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.724945    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/097a37e6-fa47-4514-b43d-5c462533947b-istiod-ca-cert\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj\" (UID: \"097a37e6-fa47-4514-b43d-5c462533947b\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj"
Oct 30 19:39:56.725100 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.725003    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/097a37e6-fa47-4514-b43d-5c462533947b-workload-socket\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj\" (UID: \"097a37e6-fa47-4514-b43d-5c462533947b\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj"
Oct 30 19:39:56.725100 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.725027    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/097a37e6-fa47-4514-b43d-5c462533947b-istio-envoy\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj\" (UID: \"097a37e6-fa47-4514-b43d-5c462533947b\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj"
Oct 30 19:39:56.725271 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.725114    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/097a37e6-fa47-4514-b43d-5c462533947b-istio-data\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj\" (UID: \"097a37e6-fa47-4514-b43d-5c462533947b\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj"
Oct 30 19:39:56.725271 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.725156    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/097a37e6-fa47-4514-b43d-5c462533947b-credential-socket\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj\" (UID: \"097a37e6-fa47-4514-b43d-5c462533947b\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj"
Oct 30 19:39:56.725271 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.725206    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/097a37e6-fa47-4514-b43d-5c462533947b-istio-podinfo\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj\" (UID: \"097a37e6-fa47-4514-b43d-5c462533947b\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj"
Oct 30 19:39:56.725433 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.725309    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9mfw\" (UniqueName: \"kubernetes.io/projected/097a37e6-fa47-4514-b43d-5c462533947b-kube-api-access-l9mfw\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj\" (UID: \"097a37e6-fa47-4514-b43d-5c462533947b\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj"
Oct 30 19:39:56.725433 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.725378    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/097a37e6-fa47-4514-b43d-5c462533947b-istio-token\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj\" (UID: \"097a37e6-fa47-4514-b43d-5c462533947b\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj"
Oct 30 19:39:56.725433 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.725399    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/097a37e6-fa47-4514-b43d-5c462533947b-workload-certs\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj\" (UID: \"097a37e6-fa47-4514-b43d-5c462533947b\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj"
Oct 30 19:39:56.725605 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.725534    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/097a37e6-fa47-4514-b43d-5c462533947b-workload-socket\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj\" (UID: \"097a37e6-fa47-4514-b43d-5c462533947b\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj"
Oct 30 19:39:56.725804 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.725773    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/097a37e6-fa47-4514-b43d-5c462533947b-credential-socket\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj\" (UID: \"097a37e6-fa47-4514-b43d-5c462533947b\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj"
Oct 30 19:39:56.725915 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.725847    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/097a37e6-fa47-4514-b43d-5c462533947b-istiod-ca-cert\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj\" (UID: \"097a37e6-fa47-4514-b43d-5c462533947b\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj"
Oct 30 19:39:56.726188 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.726133    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/097a37e6-fa47-4514-b43d-5c462533947b-istio-data\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj\" (UID: \"097a37e6-fa47-4514-b43d-5c462533947b\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj"
Oct 30 19:39:56.727890 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.727869    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/097a37e6-fa47-4514-b43d-5c462533947b-istio-envoy\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj\" (UID: \"097a37e6-fa47-4514-b43d-5c462533947b\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj"
Oct 30 19:39:56.728234 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.728211    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/097a37e6-fa47-4514-b43d-5c462533947b-istio-podinfo\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj\" (UID: \"097a37e6-fa47-4514-b43d-5c462533947b\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj"
Oct 30 19:39:56.733676 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.733652    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/097a37e6-fa47-4514-b43d-5c462533947b-istio-token\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj\" (UID: \"097a37e6-fa47-4514-b43d-5c462533947b\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj"
Oct 30 19:39:56.733832 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.733798    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9mfw\" (UniqueName: \"kubernetes.io/projected/097a37e6-fa47-4514-b43d-5c462533947b-kube-api-access-l9mfw\") pod \"e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj\" (UID: \"097a37e6-fa47-4514-b43d-5c462533947b\") " pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj"
Oct 30 19:39:56.915550 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.915511    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-156838-79f475dcd-x2qg9"
Oct 30 19:39:56.915550 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.915527    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-156838-79f475dcd-x2qg9" event={"ID":"9db7ea84-934d-41d6-aa27-c163dfc862c5","Type":"ContainerDied","Data":"873a0fbeb8c947a048c7b97a9b025ae5406b4c3833fb527d00479a0a6cd591b2"}
Oct 30 19:39:56.915807 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.915597    5043 scope.go:122] "RemoveContainer" containerID="0581936c34a1300d468eeadbd7c5aa7cfe092626c6db1d6c943d30283cd7f9cb"
Oct 30 19:39:56.917874 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.917498    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-156838-6749758ddc-tnv4d"
Oct 30 19:39:56.917874 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.917600    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-156838-6749758ddc-tnv4d" event={"ID":"21041d00-c41b-4b80-839f-c48c0f6988c7","Type":"ContainerDied","Data":"14cd2ba10157eb798f3d20c6c794a75cdc5bb0fccb119e6c0e0356274e2b0c6a"}
Oct 30 19:39:56.921107 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.921085    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj"
Oct 30 19:39:56.921893 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.921875    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd"
Oct 30 19:39:56.931061 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.931036    5043 scope.go:122] "RemoveContainer" containerID="21054e68e815c5ae4fd338a67895e5d5f3c58e0ef4d5c977521c9421d176a553"
Oct 30 19:39:56.969595 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.969532    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-156838-6749758ddc-tnv4d"]
Oct 30 19:39:56.977376 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.977167    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-156838-6749758ddc-tnv4d"]
Oct 30 19:39:56.987524 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.986840    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-156838-79f475dcd-x2qg9"]
Oct 30 19:39:56.995267 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:56.995236    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-156838-79f475dcd-x2qg9"]
Oct 30 19:39:57.240665 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:57.240638    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj"]
Oct 30 19:39:57.241319 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:39:57.241295    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod097a37e6_fa47_4514_b43d_5c462533947b.slice/crio-0882398f55ff8bb84f2a51ff5801e6da8ad5cfb35c03ed55a627d8cfc8a343a1 WatchSource:0}: Error finding container 0882398f55ff8bb84f2a51ff5801e6da8ad5cfb35c03ed55a627d8cfc8a343a1: Status 404 returned error can't find the container with id 0882398f55ff8bb84f2a51ff5801e6da8ad5cfb35c03ed55a627d8cfc8a343a1
Oct 30 19:39:57.244922 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:57.244887    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:39:57.245028 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:57.244975    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:39:57.245112 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:57.245025    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:39:57.942036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:57.941993    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj" event={"ID":"097a37e6-fa47-4514-b43d-5c462533947b","Type":"ContainerStarted","Data":"0d8116adb542a69bb69ceaba30af99c1caedb9f5dacf472d117b787d3714c39b"}
Oct 30 19:39:57.942553 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:57.942091    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj" event={"ID":"097a37e6-fa47-4514-b43d-5c462533947b","Type":"ContainerStarted","Data":"0882398f55ff8bb84f2a51ff5801e6da8ad5cfb35c03ed55a627d8cfc8a343a1"}
Oct 30 19:39:57.989815 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:57.977790    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj" podStartSLOduration=1.977770222 podStartE2EDuration="1.977770222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:57.963322888 +0000 UTC m=+1417.474382854" watchObservedRunningTime="2026-10-30 19:39:57.977770222 +0000 UTC m=+1417.488830187"
Oct 30 19:39:58.330879 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:58.330832    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ac37c8-69dbd9fb6d-xjzdf"
Oct 30 19:39:58.332709 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:58.332679    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ac37c8-57cb47d8c-4cprd"
Oct 30 19:39:58.706665 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:58.706573    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21041d00-c41b-4b80-839f-c48c0f6988c7" path="/var/lib/kubelet/pods/21041d00-c41b-4b80-839f-c48c0f6988c7/volumes"
Oct 30 19:39:58.707155 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:58.707134    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9db7ea84-934d-41d6-aa27-c163dfc862c5" path="/var/lib/kubelet/pods/9db7ea84-934d-41d6-aa27-c163dfc862c5/volumes"
Oct 30 19:39:58.922615 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:58.922588    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj"
Oct 30 19:39:58.923991 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:58.923961    5043 patch_prober.go:29] interesting pod/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Oct 30 19:39:58.924139 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:58.924044    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj" podUID="097a37e6-fa47-4514-b43d-5c462533947b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Oct 30 19:39:59.922641 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:59.922609    5043 patch_prober.go:29] interesting pod/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Oct 30 19:39:59.923073 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:39:59.922668    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj" podUID="097a37e6-fa47-4514-b43d-5c462533947b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Oct 30 19:40:00.927781 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:00.923922    5043 patch_prober.go:29] interesting pod/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Oct 30 19:40:00.927781 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:00.924100    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj" podUID="097a37e6-fa47-4514-b43d-5c462533947b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Oct 30 19:40:01.553282 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.553256    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t"
Oct 30 19:40:01.579761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.556249    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t"
Oct 30 19:40:01.590982 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.590939    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88d33c34-9387-4234-b953-446167850092-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88d33c34-9387-4234-b953-446167850092-istiod-ca-cert\") pod \"88d33c34-9387-4234-b953-446167850092\" (UID: \"88d33c34-9387-4234-b953-446167850092\") "
Oct 30 19:40:01.591165 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.591005    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88d33c34-9387-4234-b953-446167850092-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88d33c34-9387-4234-b953-446167850092-istio-envoy\") pod \"88d33c34-9387-4234-b953-446167850092\" (UID: \"88d33c34-9387-4234-b953-446167850092\") "
Oct 30 19:40:01.591165 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.591032    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88d33c34-9387-4234-b953-446167850092-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88d33c34-9387-4234-b953-446167850092-istio-podinfo\") pod \"88d33c34-9387-4234-b953-446167850092\" (UID: \"88d33c34-9387-4234-b953-446167850092\") "
Oct 30 19:40:01.591165 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.591057    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4897f8f5-579f-45ed-a964-f2379eb52848-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4897f8f5-579f-45ed-a964-f2379eb52848-istio-podinfo\") pod \"4897f8f5-579f-45ed-a964-f2379eb52848\" (UID: \"4897f8f5-579f-45ed-a964-f2379eb52848\") "
Oct 30 19:40:01.591165 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.591122    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4897f8f5-579f-45ed-a964-f2379eb52848-kube-api-access-dw2dp\" (UniqueName: \"kubernetes.io/projected/4897f8f5-579f-45ed-a964-f2379eb52848-kube-api-access-dw2dp\") pod \"4897f8f5-579f-45ed-a964-f2379eb52848\" (UID: \"4897f8f5-579f-45ed-a964-f2379eb52848\") "
Oct 30 19:40:01.591165 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.591155    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4897f8f5-579f-45ed-a964-f2379eb52848-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4897f8f5-579f-45ed-a964-f2379eb52848-workload-certs\") pod \"4897f8f5-579f-45ed-a964-f2379eb52848\" (UID: \"4897f8f5-579f-45ed-a964-f2379eb52848\") "
Oct 30 19:40:01.591367 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.591187    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4897f8f5-579f-45ed-a964-f2379eb52848-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4897f8f5-579f-45ed-a964-f2379eb52848-credential-socket\") pod \"4897f8f5-579f-45ed-a964-f2379eb52848\" (UID: \"4897f8f5-579f-45ed-a964-f2379eb52848\") "
Oct 30 19:40:01.591367 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.591220    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88d33c34-9387-4234-b953-446167850092-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88d33c34-9387-4234-b953-446167850092-istio-data\") pod \"88d33c34-9387-4234-b953-446167850092\" (UID: \"88d33c34-9387-4234-b953-446167850092\") "
Oct 30 19:40:01.591367 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.591237    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4897f8f5-579f-45ed-a964-f2379eb52848-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4897f8f5-579f-45ed-a964-f2379eb52848-istio-envoy\") pod \"4897f8f5-579f-45ed-a964-f2379eb52848\" (UID: \"4897f8f5-579f-45ed-a964-f2379eb52848\") "
Oct 30 19:40:01.591367 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.591265    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88d33c34-9387-4234-b953-446167850092-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88d33c34-9387-4234-b953-446167850092-workload-socket\") pod \"88d33c34-9387-4234-b953-446167850092\" (UID: \"88d33c34-9387-4234-b953-446167850092\") "
Oct 30 19:40:01.591367 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.591306    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4897f8f5-579f-45ed-a964-f2379eb52848-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4897f8f5-579f-45ed-a964-f2379eb52848-workload-socket\") pod \"4897f8f5-579f-45ed-a964-f2379eb52848\" (UID: \"4897f8f5-579f-45ed-a964-f2379eb52848\") "
Oct 30 19:40:01.591367 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.591341    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88d33c34-9387-4234-b953-446167850092-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88d33c34-9387-4234-b953-446167850092-workload-certs\") pod \"88d33c34-9387-4234-b953-446167850092\" (UID: \"88d33c34-9387-4234-b953-446167850092\") "
Oct 30 19:40:01.591550 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.591378    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88d33c34-9387-4234-b953-446167850092-kube-api-access-2cs47\" (UniqueName: \"kubernetes.io/projected/88d33c34-9387-4234-b953-446167850092-kube-api-access-2cs47\") pod \"88d33c34-9387-4234-b953-446167850092\" (UID: \"88d33c34-9387-4234-b953-446167850092\") "
Oct 30 19:40:01.591550 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.591451    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88d33c34-9387-4234-b953-446167850092-istio-token\" (UniqueName: \"kubernetes.io/projected/88d33c34-9387-4234-b953-446167850092-istio-token\") pod \"88d33c34-9387-4234-b953-446167850092\" (UID: \"88d33c34-9387-4234-b953-446167850092\") "
Oct 30 19:40:01.591550 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.591491    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4897f8f5-579f-45ed-a964-f2379eb52848-istio-token\" (UniqueName: \"kubernetes.io/projected/4897f8f5-579f-45ed-a964-f2379eb52848-istio-token\") pod \"4897f8f5-579f-45ed-a964-f2379eb52848\" (UID: \"4897f8f5-579f-45ed-a964-f2379eb52848\") "
Oct 30 19:40:01.591550 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.591512    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4897f8f5-579f-45ed-a964-f2379eb52848-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4897f8f5-579f-45ed-a964-f2379eb52848-istiod-ca-cert\") pod \"4897f8f5-579f-45ed-a964-f2379eb52848\" (UID: \"4897f8f5-579f-45ed-a964-f2379eb52848\") "
Oct 30 19:40:01.591550 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.591534    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4897f8f5-579f-45ed-a964-f2379eb52848-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4897f8f5-579f-45ed-a964-f2379eb52848-istio-data\") pod \"4897f8f5-579f-45ed-a964-f2379eb52848\" (UID: \"4897f8f5-579f-45ed-a964-f2379eb52848\") "
Oct 30 19:40:01.591715 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.591573    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88d33c34-9387-4234-b953-446167850092-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88d33c34-9387-4234-b953-446167850092-credential-socket\") pod \"88d33c34-9387-4234-b953-446167850092\" (UID: \"88d33c34-9387-4234-b953-446167850092\") "
Oct 30 19:40:01.592796 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.592634    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88d33c34-9387-4234-b953-446167850092-credential-socket" pod "88d33c34-9387-4234-b953-446167850092" (UID: "88d33c34-9387-4234-b953-446167850092"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:01.592796 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.592665    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4897f8f5-579f-45ed-a964-f2379eb52848-workload-socket" pod "4897f8f5-579f-45ed-a964-f2379eb52848" (UID: "4897f8f5-579f-45ed-a964-f2379eb52848"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:01.593076 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.592913    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88d33c34-9387-4234-b953-446167850092-workload-socket" pod "88d33c34-9387-4234-b953-446167850092" (UID: "88d33c34-9387-4234-b953-446167850092"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:01.593076 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.592976    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4897f8f5-579f-45ed-a964-f2379eb52848-istiod-ca-cert" pod "4897f8f5-579f-45ed-a964-f2379eb52848" (UID: "4897f8f5-579f-45ed-a964-f2379eb52848"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:40:01.593382 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.593114    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88d33c34-9387-4234-b953-446167850092-workload-certs" pod "88d33c34-9387-4234-b953-446167850092" (UID: "88d33c34-9387-4234-b953-446167850092"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:01.593382 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.593301    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4897f8f5-579f-45ed-a964-f2379eb52848-workload-certs" pod "4897f8f5-579f-45ed-a964-f2379eb52848" (UID: "4897f8f5-579f-45ed-a964-f2379eb52848"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:01.593537 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.593431    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4897f8f5-579f-45ed-a964-f2379eb52848-istio-data" pod "4897f8f5-579f-45ed-a964-f2379eb52848" (UID: "4897f8f5-579f-45ed-a964-f2379eb52848"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:01.593648 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.593623    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4897f8f5-579f-45ed-a964-f2379eb52848-credential-socket" pod "4897f8f5-579f-45ed-a964-f2379eb52848" (UID: "4897f8f5-579f-45ed-a964-f2379eb52848"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:01.593946 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.593922    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88d33c34-9387-4234-b953-446167850092-istiod-ca-cert" pod "88d33c34-9387-4234-b953-446167850092" (UID: "88d33c34-9387-4234-b953-446167850092"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:40:01.594296 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.594253    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88d33c34-9387-4234-b953-446167850092-istio-data" pod "88d33c34-9387-4234-b953-446167850092" (UID: "88d33c34-9387-4234-b953-446167850092"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:01.599024 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.598831    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4897f8f5-579f-45ed-a964-f2379eb52848-kube-api-access-dw2dp" pod "4897f8f5-579f-45ed-a964-f2379eb52848" (UID: "4897f8f5-579f-45ed-a964-f2379eb52848"). InnerVolumeSpecName "kube-api-access-dw2dp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:40:01.600887 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.599970    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4897f8f5-579f-45ed-a964-f2379eb52848-istio-token" pod "4897f8f5-579f-45ed-a964-f2379eb52848" (UID: "4897f8f5-579f-45ed-a964-f2379eb52848"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:40:01.600887 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.600065    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88d33c34-9387-4234-b953-446167850092-istio-token" pod "88d33c34-9387-4234-b953-446167850092" (UID: "88d33c34-9387-4234-b953-446167850092"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:40:01.600887 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.600262    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4897f8f5-579f-45ed-a964-f2379eb52848-istio-envoy" pod "4897f8f5-579f-45ed-a964-f2379eb52848" (UID: "4897f8f5-579f-45ed-a964-f2379eb52848"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:01.601183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.601159    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88d33c34-9387-4234-b953-446167850092-kube-api-access-2cs47" pod "88d33c34-9387-4234-b953-446167850092" (UID: "88d33c34-9387-4234-b953-446167850092"). InnerVolumeSpecName "kube-api-access-2cs47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:40:01.602277 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.602241    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4897f8f5-579f-45ed-a964-f2379eb52848-istio-podinfo" pod "4897f8f5-579f-45ed-a964-f2379eb52848" (UID: "4897f8f5-579f-45ed-a964-f2379eb52848"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:40:01.602565 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.602542    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88d33c34-9387-4234-b953-446167850092-istio-podinfo" pod "88d33c34-9387-4234-b953-446167850092" (UID: "88d33c34-9387-4234-b953-446167850092"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:40:01.603272 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.603243    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88d33c34-9387-4234-b953-446167850092-istio-envoy" pod "88d33c34-9387-4234-b953-446167850092" (UID: "88d33c34-9387-4234-b953-446167850092"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:01.694055 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.694021    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88d33c34-9387-4234-b953-446167850092-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:01.694055 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.694051    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88d33c34-9387-4234-b953-446167850092-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:01.694055 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.694062    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4897f8f5-579f-45ed-a964-f2379eb52848-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:01.694398 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.694071    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw2dp\" (UniqueName: \"kubernetes.io/projected/4897f8f5-579f-45ed-a964-f2379eb52848-kube-api-access-dw2dp\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:01.694398 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.694082    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4897f8f5-579f-45ed-a964-f2379eb52848-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:01.694398 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.694091    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4897f8f5-579f-45ed-a964-f2379eb52848-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:01.694398 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.694100    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88d33c34-9387-4234-b953-446167850092-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:01.694398 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.694109    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4897f8f5-579f-45ed-a964-f2379eb52848-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:01.694398 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.694117    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88d33c34-9387-4234-b953-446167850092-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:01.694398 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.694125    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4897f8f5-579f-45ed-a964-f2379eb52848-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:01.694398 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.694135    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88d33c34-9387-4234-b953-446167850092-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:01.694398 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.694144    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cs47\" (UniqueName: \"kubernetes.io/projected/88d33c34-9387-4234-b953-446167850092-kube-api-access-2cs47\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:01.694398 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.694154    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88d33c34-9387-4234-b953-446167850092-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:01.694398 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.694162    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4897f8f5-579f-45ed-a964-f2379eb52848-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:01.694398 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.694170    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4897f8f5-579f-45ed-a964-f2379eb52848-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:01.694398 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.694178    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4897f8f5-579f-45ed-a964-f2379eb52848-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:01.694398 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.694186    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88d33c34-9387-4234-b953-446167850092-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:01.694398 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.694194    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88d33c34-9387-4234-b953-446167850092-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:01.921633 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.921531    5043 patch_prober.go:29] interesting pod/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Oct 30 19:40:01.921633 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:01.921598    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj" podUID="097a37e6-fa47-4514-b43d-5c462533947b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Oct 30 19:40:02.097883 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:02.097135    5043 generic.go:356] "Generic (PLEG): container finished" podID="4897f8f5-579f-45ed-a964-f2379eb52848" containerID="ed9f07518d42efa6ff32cfd0e47b68eded8fba60afb242e22f90fcffff274993" exitCode=0
Oct 30 19:40:02.097883 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:02.097229    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t" event={"ID":"4897f8f5-579f-45ed-a964-f2379eb52848","Type":"ContainerDied","Data":"ed9f07518d42efa6ff32cfd0e47b68eded8fba60afb242e22f90fcffff274993"}
Oct 30 19:40:02.097883 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:02.097301    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t" event={"ID":"4897f8f5-579f-45ed-a964-f2379eb52848","Type":"ContainerDied","Data":"34b9a8c03add878837b79d96240d8b7347b23f3164cec37e50bb8a9c34a47e8d"}
Oct 30 19:40:02.097883 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:02.097330    5043 scope.go:122] "RemoveContainer" containerID="ed9f07518d42efa6ff32cfd0e47b68eded8fba60afb242e22f90fcffff274993"
Oct 30 19:40:02.097883 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:02.097684    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t"
Oct 30 19:40:02.099692 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:02.099667    5043 generic.go:356] "Generic (PLEG): container finished" podID="88d33c34-9387-4234-b953-446167850092" containerID="53a012b0030f36e1b61ed75f81affb415ca4eaa1f4182837fd73d5322d6c7752" exitCode=0
Oct 30 19:40:02.099824 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:02.099705    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t" event={"ID":"88d33c34-9387-4234-b953-446167850092","Type":"ContainerDied","Data":"53a012b0030f36e1b61ed75f81affb415ca4eaa1f4182837fd73d5322d6c7752"}
Oct 30 19:40:02.099824 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:02.099803    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t" event={"ID":"88d33c34-9387-4234-b953-446167850092","Type":"ContainerDied","Data":"1612eb5a5bf67685dba8a0b903a054bb547030b8ad68eeacc0b016644b655e36"}
Oct 30 19:40:02.100017 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:02.099999    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t"
Oct 30 19:40:02.112852 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:02.112829    5043 scope.go:122] "RemoveContainer" containerID="ed9f07518d42efa6ff32cfd0e47b68eded8fba60afb242e22f90fcffff274993"
Oct 30 19:40:02.113134 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:40:02.113114    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed9f07518d42efa6ff32cfd0e47b68eded8fba60afb242e22f90fcffff274993\": container with ID starting with ed9f07518d42efa6ff32cfd0e47b68eded8fba60afb242e22f90fcffff274993 not found: ID does not exist" containerID="ed9f07518d42efa6ff32cfd0e47b68eded8fba60afb242e22f90fcffff274993"
Oct 30 19:40:02.113215 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:02.113145    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed9f07518d42efa6ff32cfd0e47b68eded8fba60afb242e22f90fcffff274993"} err="failed to get container status \"ed9f07518d42efa6ff32cfd0e47b68eded8fba60afb242e22f90fcffff274993\": rpc error: code = NotFound desc = could not find container \"ed9f07518d42efa6ff32cfd0e47b68eded8fba60afb242e22f90fcffff274993\": container with ID starting with ed9f07518d42efa6ff32cfd0e47b68eded8fba60afb242e22f90fcffff274993 not found: ID does not exist"
Oct 30 19:40:02.113215 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:02.113167    5043 scope.go:122] "RemoveContainer" containerID="53a012b0030f36e1b61ed75f81affb415ca4eaa1f4182837fd73d5322d6c7752"
Oct 30 19:40:02.124350 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:02.124333    5043 scope.go:122] "RemoveContainer" containerID="53a012b0030f36e1b61ed75f81affb415ca4eaa1f4182837fd73d5322d6c7752"
Oct 30 19:40:02.124626 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:40:02.124599    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53a012b0030f36e1b61ed75f81affb415ca4eaa1f4182837fd73d5322d6c7752\": container with ID starting with 53a012b0030f36e1b61ed75f81affb415ca4eaa1f4182837fd73d5322d6c7752 not found: ID does not exist" containerID="53a012b0030f36e1b61ed75f81affb415ca4eaa1f4182837fd73d5322d6c7752"
Oct 30 19:40:02.124676 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:02.124636    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53a012b0030f36e1b61ed75f81affb415ca4eaa1f4182837fd73d5322d6c7752"} err="failed to get container status \"53a012b0030f36e1b61ed75f81affb415ca4eaa1f4182837fd73d5322d6c7752\": rpc error: code = NotFound desc = could not find container \"53a012b0030f36e1b61ed75f81affb415ca4eaa1f4182837fd73d5322d6c7752\": container with ID starting with 53a012b0030f36e1b61ed75f81affb415ca4eaa1f4182837fd73d5322d6c7752 not found: ID does not exist"
Oct 30 19:40:02.134474 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:02.134450    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t"]
Oct 30 19:40:02.139616 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:02.139594    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-28g9t"]
Oct 30 19:40:02.148321 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:02.148297    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t"]
Oct 30 19:40:02.154761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:02.154724    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-npw9t"]
Oct 30 19:40:02.699568 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:02.699494    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4897f8f5-579f-45ed-a964-f2379eb52848" path="/var/lib/kubelet/pods/4897f8f5-579f-45ed-a964-f2379eb52848/volumes"
Oct 30 19:40:02.700413 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:02.700388    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88d33c34-9387-4234-b953-446167850092" path="/var/lib/kubelet/pods/88d33c34-9387-4234-b953-446167850092/volumes"
Oct 30 19:40:02.930092 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:02.930040    5043 patch_prober.go:29] interesting pod/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Oct 30 19:40:02.930614 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:02.930576    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj" podUID="097a37e6-fa47-4514-b43d-5c462533947b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Oct 30 19:40:03.935280 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:03.935246    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj"
Oct 30 19:40:03.936001 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:03.935786    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj"
Oct 30 19:40:03.936945 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:03.936924    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj"
Oct 30 19:40:11.843306 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:11.843210    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-156838-ljhgm"]
Oct 30 19:40:11.850028 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:11.849979    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-156838-ljhgm"]
Oct 30 19:40:12.647409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:12.647373    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07ec9d46-74d0-4638-bdbd-67f286deb315" path="/var/lib/kubelet/pods/07ec9d46-74d0-4638-bdbd-67f286deb315/volumes"
Oct 30 19:40:14.148799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:14.148767    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d1d256-svs6j"]
Oct 30 19:40:14.153817 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:14.153790    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d1d256-svs6j"]
Oct 30 19:40:14.646725 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:14.646698    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad79a09b-6291-4c55-b9a5-cd0f888a1557" path="/var/lib/kubelet/pods/ad79a09b-6291-4c55-b9a5-cd0f888a1557/volumes"
Oct 30 19:40:22.139757 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:22.137539    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd"]
Oct 30 19:40:22.139757 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:22.137824    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd" podUID="2db29833-fd7c-4dfd-88ba-6ae264d9f8b5" containerName="istio-proxy" containerID="cri-o://93d64382b6282c4799809e0405a91470ae3f723232661dcb860d85da7b7b62e4" gracePeriod=30
Oct 30 19:40:23.344686 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:23.344650    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp"]
Oct 30 19:40:23.345608 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:23.345552    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp" podUID="a74dccf1-3de7-4f70-a549-281361992615" containerName="istio-proxy" containerID="cri-o://a9ba0d5574dfb6107ce27928d476f6e1716cc4ae0765f3742efbf5c3ce01a001" gracePeriod=30
Oct 30 19:40:23.949541 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:23.946885    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-22af66-5klt4"]
Oct 30 19:40:23.949541 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:23.948807    5043 cpu_manager.go:414] "containerMap: removing container" podUID="88d33c34-9387-4234-b953-446167850092" containerName="istio-proxy"
Oct 30 19:40:23.949541 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:23.948829    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88d33c34-9387-4234-b953-446167850092" containerName="istio-proxy"
Oct 30 19:40:23.949541 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:23.948944    5043 cpu_manager.go:414] "containerMap: removing container" podUID="4897f8f5-579f-45ed-a964-f2379eb52848" containerName="istio-proxy"
Oct 30 19:40:23.949541 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:23.948954    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4897f8f5-579f-45ed-a964-f2379eb52848" containerName="istio-proxy"
Oct 30 19:40:23.949541 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:23.949390    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="88d33c34-9387-4234-b953-446167850092" containerName="istio-proxy"
Oct 30 19:40:23.949541 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:23.949428    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="4897f8f5-579f-45ed-a964-f2379eb52848" containerName="istio-proxy"
Oct 30 19:40:23.999219 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:23.999179    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-22af66-5klt4"]
Oct 30 19:40:23.999573 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:23.999552    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-22af66-5klt4"
Oct 30 19:40:24.009005 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:24.008977    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:40:24.009273 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:24.009259    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:40:24.017215 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:24.015670    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:40:24.017215 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:24.015726    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:40:24.039244 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:24.039210    5043 scope.go:122] "RemoveContainer" containerID="b3a9ccf5a1557f239927d2e47df86d9e2e5ce0d03b5dd22f0164b00fcab8315b"
Oct 30 19:40:24.051769 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:24.051726    5043 scope.go:122] "RemoveContainer" containerID="e712f16412844fb2165ce8b2c0e2ee7af346e773acd17f690f2668e3ebbee758"
Oct 30 19:40:24.098042 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:24.098011    5043 scope.go:122] "RemoveContainer" containerID="309c4ffdcd53aed1c8eece2f7d35cc3362b784a2802bcc549b3e38f89f1d6fc3"
Oct 30 19:40:24.121588 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:24.121549    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7b8ba26-6bb8-4666-b83f-808dc079edef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-22af66-5klt4\" (UID: \"a7b8ba26-6bb8-4666-b83f-808dc079edef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-22af66-5klt4"
Oct 30 19:40:24.121755 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:24.121684    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-469rx\" (UniqueName: \"kubernetes.io/projected/a7b8ba26-6bb8-4666-b83f-808dc079edef-kube-api-access-469rx\") pod \"maas-api-revoke-keys-e2e-api-b-22af66-5klt4\" (UID: \"a7b8ba26-6bb8-4666-b83f-808dc079edef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-22af66-5klt4"
Oct 30 19:40:24.122962 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:24.122927    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7b8ba26-6bb8-4666-b83f-808dc079edef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-22af66-5klt4\" (UID: \"a7b8ba26-6bb8-4666-b83f-808dc079edef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-22af66-5klt4"
Oct 30 19:40:24.224551 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:24.224525    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7b8ba26-6bb8-4666-b83f-808dc079edef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-22af66-5klt4\" (UID: \"a7b8ba26-6bb8-4666-b83f-808dc079edef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-22af66-5klt4"
Oct 30 19:40:24.224853 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:24.224819    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7b8ba26-6bb8-4666-b83f-808dc079edef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-22af66-5klt4\" (UID: \"a7b8ba26-6bb8-4666-b83f-808dc079edef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-22af66-5klt4"
Oct 30 19:40:24.225052 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:24.225037    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-469rx\" (UniqueName: \"kubernetes.io/projected/a7b8ba26-6bb8-4666-b83f-808dc079edef-kube-api-access-469rx\") pod \"maas-api-revoke-keys-e2e-api-b-22af66-5klt4\" (UID: \"a7b8ba26-6bb8-4666-b83f-808dc079edef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-22af66-5klt4"
Oct 30 19:40:24.226518 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:24.226478    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7b8ba26-6bb8-4666-b83f-808dc079edef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-22af66-5klt4\" (UID: \"a7b8ba26-6bb8-4666-b83f-808dc079edef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-22af66-5klt4"
Oct 30 19:40:24.235396 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:24.234669    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7b8ba26-6bb8-4666-b83f-808dc079edef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-22af66-5klt4\" (UID: \"a7b8ba26-6bb8-4666-b83f-808dc079edef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-22af66-5klt4"
Oct 30 19:40:24.237525 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:24.237506    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-469rx\" (UniqueName: \"kubernetes.io/projected/a7b8ba26-6bb8-4666-b83f-808dc079edef-kube-api-access-469rx\") pod \"maas-api-revoke-keys-e2e-api-b-22af66-5klt4\" (UID: \"a7b8ba26-6bb8-4666-b83f-808dc079edef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-22af66-5klt4"
Oct 30 19:40:24.335124 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:24.335086    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-22af66-5klt4"
Oct 30 19:40:24.711809 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:24.711779    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-22af66-5klt4"]
Oct 30 19:40:24.712408 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:40:24.712385    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7b8ba26_6bb8_4666_b83f_808dc079edef.slice/crio-7df3990178751cb0200228dac5add55e542c256a5f8aabec0f8f17d5035fdcb2 WatchSource:0}: Error finding container 7df3990178751cb0200228dac5add55e542c256a5f8aabec0f8f17d5035fdcb2: Status 404 returned error can't find the container with id 7df3990178751cb0200228dac5add55e542c256a5f8aabec0f8f17d5035fdcb2
Oct 30 19:40:24.988850 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:40:24.988704    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:40:25.016340 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:25.016306    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-22af66-5klt4" event={"ID":"a7b8ba26-6bb8-4666-b83f-808dc079edef","Type":"ContainerStarted","Data":"7df3990178751cb0200228dac5add55e542c256a5f8aabec0f8f17d5035fdcb2"}
Oct 30 19:40:26.031447 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:26.031410    5043 generic.go:356] "Generic (PLEG): container finished" podID="a7b8ba26-6bb8-4666-b83f-808dc079edef" containerID="51d20eb0846e85304fb37875a43e06632930dcbfb742dcc4a2ba4e1986c85b34" exitCode=0
Oct 30 19:40:26.031956 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:26.031551    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-22af66-5klt4" event={"ID":"a7b8ba26-6bb8-4666-b83f-808dc079edef","Type":"ContainerDied","Data":"51d20eb0846e85304fb37875a43e06632930dcbfb742dcc4a2ba4e1986c85b34"}
Oct 30 19:40:27.345989 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:27.345955    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-22af66-5klt4"
Oct 30 19:40:27.499061 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:27.498067    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7b8ba26-6bb8-4666-b83f-808dc079edef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7b8ba26-6bb8-4666-b83f-808dc079edef-maas-api-openshift-ca\") pod \"a7b8ba26-6bb8-4666-b83f-808dc079edef\" (UID: \"a7b8ba26-6bb8-4666-b83f-808dc079edef\") "
Oct 30 19:40:27.499061 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:27.498304    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7b8ba26-6bb8-4666-b83f-808dc079edef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7b8ba26-6bb8-4666-b83f-808dc079edef-maas-api-xks-ca\") pod \"a7b8ba26-6bb8-4666-b83f-808dc079edef\" (UID: \"a7b8ba26-6bb8-4666-b83f-808dc079edef\") "
Oct 30 19:40:27.499061 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:27.498353    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7b8ba26-6bb8-4666-b83f-808dc079edef-kube-api-access-469rx\" (UniqueName: \"kubernetes.io/projected/a7b8ba26-6bb8-4666-b83f-808dc079edef-kube-api-access-469rx\") pod \"a7b8ba26-6bb8-4666-b83f-808dc079edef\" (UID: \"a7b8ba26-6bb8-4666-b83f-808dc079edef\") "
Oct 30 19:40:27.499061 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:27.498489    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7b8ba26-6bb8-4666-b83f-808dc079edef-maas-api-openshift-ca" pod "a7b8ba26-6bb8-4666-b83f-808dc079edef" (UID: "a7b8ba26-6bb8-4666-b83f-808dc079edef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:40:27.499859 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:27.499824    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7b8ba26-6bb8-4666-b83f-808dc079edef-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:27.502406 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:27.502369    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7b8ba26-6bb8-4666-b83f-808dc079edef-maas-api-xks-ca" pod "a7b8ba26-6bb8-4666-b83f-808dc079edef" (UID: "a7b8ba26-6bb8-4666-b83f-808dc079edef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:40:27.513177 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:27.513152    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7b8ba26-6bb8-4666-b83f-808dc079edef-kube-api-access-469rx" pod "a7b8ba26-6bb8-4666-b83f-808dc079edef" (UID: "a7b8ba26-6bb8-4666-b83f-808dc079edef"). InnerVolumeSpecName "kube-api-access-469rx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:40:27.601125 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:27.601041    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7b8ba26-6bb8-4666-b83f-808dc079edef-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:27.601125 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:27.601071    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-469rx\" (UniqueName: \"kubernetes.io/projected/a7b8ba26-6bb8-4666-b83f-808dc079edef-kube-api-access-469rx\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:27.651960 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:27.651927    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd"
Oct 30 19:40:27.813828 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:27.811335    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-credential-socket\") pod \"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5\" (UID: \"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5\") "
Oct 30 19:40:27.813828 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:27.811495    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-istio-token\" (UniqueName: \"kubernetes.io/projected/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-istio-token\") pod \"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5\" (UID: \"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5\") "
Oct 30 19:40:27.813828 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:27.811586    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-istio-podinfo\") pod \"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5\" (UID: \"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5\") "
Oct 30 19:40:27.813828 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:27.811648    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-workload-certs\") pod \"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5\" (UID: \"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5\") "
Oct 30 19:40:27.813828 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:27.811703    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-istio-data\") pod \"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5\" (UID: \"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5\") "
Oct 30 19:40:27.813828 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:27.811822    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-istio-envoy\") pod \"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5\" (UID: \"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5\") "
Oct 30 19:40:27.813828 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:27.811872    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-workload-socket\") pod \"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5\" (UID: \"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5\") "
Oct 30 19:40:27.813828 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:27.811941    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-kube-api-access-kkdwb\" (UniqueName: \"kubernetes.io/projected/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-kube-api-access-kkdwb\") pod \"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5\" (UID: \"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5\") "
Oct 30 19:40:27.813828 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:27.811994    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-istiod-ca-cert\") pod \"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5\" (UID: \"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5\") "
Oct 30 19:40:27.816613 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:27.814099    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-credential-socket" pod "2db29833-fd7c-4dfd-88ba-6ae264d9f8b5" (UID: "2db29833-fd7c-4dfd-88ba-6ae264d9f8b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:27.816613 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:27.815397    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-workload-socket" pod "2db29833-fd7c-4dfd-88ba-6ae264d9f8b5" (UID: "2db29833-fd7c-4dfd-88ba-6ae264d9f8b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:27.816613 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:27.815449    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-istio-data" pod "2db29833-fd7c-4dfd-88ba-6ae264d9f8b5" (UID: "2db29833-fd7c-4dfd-88ba-6ae264d9f8b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:27.816613 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:27.815660    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-workload-certs" pod "2db29833-fd7c-4dfd-88ba-6ae264d9f8b5" (UID: "2db29833-fd7c-4dfd-88ba-6ae264d9f8b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:27.816613 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:27.815929    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-istiod-ca-cert" pod "2db29833-fd7c-4dfd-88ba-6ae264d9f8b5" (UID: "2db29833-fd7c-4dfd-88ba-6ae264d9f8b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:40:27.822535 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:27.822487    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-istio-podinfo" pod "2db29833-fd7c-4dfd-88ba-6ae264d9f8b5" (UID: "2db29833-fd7c-4dfd-88ba-6ae264d9f8b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:40:27.824717 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:27.824686    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-istio-token" pod "2db29833-fd7c-4dfd-88ba-6ae264d9f8b5" (UID: "2db29833-fd7c-4dfd-88ba-6ae264d9f8b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:40:27.826665 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:27.826635    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-istio-envoy" pod "2db29833-fd7c-4dfd-88ba-6ae264d9f8b5" (UID: "2db29833-fd7c-4dfd-88ba-6ae264d9f8b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:27.826953 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:27.826929    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-kube-api-access-kkdwb" pod "2db29833-fd7c-4dfd-88ba-6ae264d9f8b5" (UID: "2db29833-fd7c-4dfd-88ba-6ae264d9f8b5"). InnerVolumeSpecName "kube-api-access-kkdwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:40:27.914061 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:27.913974    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:27.914061 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:27.914008    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:27.914061 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:27.914018    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:27.914061 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:27.914027    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:27.914061 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:27.914047    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:27.914061 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:27.914058    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:27.914448 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:27.914071    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkdwb\" (UniqueName: \"kubernetes.io/projected/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-kube-api-access-kkdwb\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:27.914448 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:27.914087    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:27.914448 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:27.914102    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:28.073432 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.073406    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-22af66-5klt4"
Oct 30 19:40:28.074063 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.073380    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-22af66-5klt4" event={"ID":"a7b8ba26-6bb8-4666-b83f-808dc079edef","Type":"ContainerDied","Data":"7df3990178751cb0200228dac5add55e542c256a5f8aabec0f8f17d5035fdcb2"}
Oct 30 19:40:28.074223 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.074093    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7df3990178751cb0200228dac5add55e542c256a5f8aabec0f8f17d5035fdcb2"
Oct 30 19:40:28.076202 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.076162    5043 generic.go:356] "Generic (PLEG): container finished" podID="2db29833-fd7c-4dfd-88ba-6ae264d9f8b5" containerID="93d64382b6282c4799809e0405a91470ae3f723232661dcb860d85da7b7b62e4" exitCode=0
Oct 30 19:40:28.076313 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.076213    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd"
Oct 30 19:40:28.076527 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.076510    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd" event={"ID":"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5","Type":"ContainerDied","Data":"93d64382b6282c4799809e0405a91470ae3f723232661dcb860d85da7b7b62e4"}
Oct 30 19:40:28.076585 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.076558    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd" event={"ID":"2db29833-fd7c-4dfd-88ba-6ae264d9f8b5","Type":"ContainerDied","Data":"d28779c9c320dd14dabb5253a1f4063ae57cb8289e4a0459019c2b35567247e3"}
Oct 30 19:40:28.076644 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.076588    5043 scope.go:122] "RemoveContainer" containerID="93d64382b6282c4799809e0405a91470ae3f723232661dcb860d85da7b7b62e4"
Oct 30 19:40:28.097559 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.096016    5043 scope.go:122] "RemoveContainer" containerID="bf77081744284b6717031a6400821be62368290fa84e5f36bf4da82dcffa73b4"
Oct 30 19:40:28.109004 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.108979    5043 scope.go:122] "RemoveContainer" containerID="93d64382b6282c4799809e0405a91470ae3f723232661dcb860d85da7b7b62e4"
Oct 30 19:40:28.109299 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:40:28.109278    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93d64382b6282c4799809e0405a91470ae3f723232661dcb860d85da7b7b62e4\": container with ID starting with 93d64382b6282c4799809e0405a91470ae3f723232661dcb860d85da7b7b62e4 not found: ID does not exist" containerID="93d64382b6282c4799809e0405a91470ae3f723232661dcb860d85da7b7b62e4"
Oct 30 19:40:28.109417 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.109314    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93d64382b6282c4799809e0405a91470ae3f723232661dcb860d85da7b7b62e4"} err="failed to get container status \"93d64382b6282c4799809e0405a91470ae3f723232661dcb860d85da7b7b62e4\": rpc error: code = NotFound desc = could not find container \"93d64382b6282c4799809e0405a91470ae3f723232661dcb860d85da7b7b62e4\": container with ID starting with 93d64382b6282c4799809e0405a91470ae3f723232661dcb860d85da7b7b62e4 not found: ID does not exist"
Oct 30 19:40:28.109417 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.109337    5043 scope.go:122] "RemoveContainer" containerID="bf77081744284b6717031a6400821be62368290fa84e5f36bf4da82dcffa73b4"
Oct 30 19:40:28.109611 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:40:28.109591    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf77081744284b6717031a6400821be62368290fa84e5f36bf4da82dcffa73b4\": container with ID starting with bf77081744284b6717031a6400821be62368290fa84e5f36bf4da82dcffa73b4 not found: ID does not exist" containerID="bf77081744284b6717031a6400821be62368290fa84e5f36bf4da82dcffa73b4"
Oct 30 19:40:28.109688 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.109618    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf77081744284b6717031a6400821be62368290fa84e5f36bf4da82dcffa73b4"} err="failed to get container status \"bf77081744284b6717031a6400821be62368290fa84e5f36bf4da82dcffa73b4\": rpc error: code = NotFound desc = could not find container \"bf77081744284b6717031a6400821be62368290fa84e5f36bf4da82dcffa73b4\": container with ID starting with bf77081744284b6717031a6400821be62368290fa84e5f36bf4da82dcffa73b4 not found: ID does not exist"
Oct 30 19:40:28.132754 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.132714    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd"]
Oct 30 19:40:28.138136 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.138112    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-156838-openshift-default-6cc4b565b5-ljxwd"]
Oct 30 19:40:28.189335 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.189251    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3e35df-7f747b7fb-gdjfs"]
Oct 30 19:40:28.190561 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.190532    5043 cpu_manager.go:414] "containerMap: removing container" podUID="2db29833-fd7c-4dfd-88ba-6ae264d9f8b5" containerName="istio-proxy"
Oct 30 19:40:28.190561 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.190561    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2db29833-fd7c-4dfd-88ba-6ae264d9f8b5" containerName="istio-proxy"
Oct 30 19:40:28.190772 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.190579    5043 cpu_manager.go:414] "containerMap: removing container" podUID="2db29833-fd7c-4dfd-88ba-6ae264d9f8b5" containerName="istio-proxy"
Oct 30 19:40:28.190772 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.190587    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2db29833-fd7c-4dfd-88ba-6ae264d9f8b5" containerName="istio-proxy"
Oct 30 19:40:28.190772 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.190606    5043 cpu_manager.go:414] "containerMap: removing container" podUID="a7b8ba26-6bb8-4666-b83f-808dc079edef" containerName="revoke-keys"
Oct 30 19:40:28.190772 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.190616    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7b8ba26-6bb8-4666-b83f-808dc079edef" containerName="revoke-keys"
Oct 30 19:40:28.191002 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.190822    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="2db29833-fd7c-4dfd-88ba-6ae264d9f8b5" containerName="istio-proxy"
Oct 30 19:40:28.191002 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.190838    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="2db29833-fd7c-4dfd-88ba-6ae264d9f8b5" containerName="istio-proxy"
Oct 30 19:40:28.191002 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.190852    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7b8ba26-6bb8-4666-b83f-808dc079edef" containerName="revoke-keys"
Oct 30 19:40:28.263366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.263331    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3e35df-7874fb7595-2v6xk"]
Oct 30 19:40:28.269108 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.265280    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3e35df-7f747b7fb-gdjfs"
Oct 30 19:40:28.270165 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.269685    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-3e35df\""
Oct 30 19:40:28.311469 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.310097    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-3e35df-dockercfg-zhldk\""
Oct 30 19:40:28.459810 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.431955    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3e35df-7f747b7fb-gdjfs"]
Oct 30 19:40:28.459810 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.432011    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3e35df-7874fb7595-2v6xk"]
Oct 30 19:40:28.459810 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.432125    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3e35df-7874fb7595-2v6xk"
Oct 30 19:40:28.511760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.511563    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtzw5\" (UniqueName: \"kubernetes.io/projected/a47d8311-eef7-47b2-b426-140198d456ee-kube-api-access-jtzw5\") pod \"payload-pre-processing-e2e-inf-a-3e35df-7874fb7595-2v6xk\" (UID: \"a47d8311-eef7-47b2-b426-140198d456ee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3e35df-7874fb7595-2v6xk"
Oct 30 19:40:28.511760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.511658    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97fbcbc0-ac3f-4957-a0f3-a6de47fb4a15-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3e35df-7f747b7fb-gdjfs\" (UID: \"97fbcbc0-ac3f-4957-a0f3-a6de47fb4a15\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3e35df-7f747b7fb-gdjfs"
Oct 30 19:40:28.511760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.511687    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z9kv\" (UniqueName: \"kubernetes.io/projected/97fbcbc0-ac3f-4957-a0f3-a6de47fb4a15-kube-api-access-2z9kv\") pod \"payload-processing-e2e-inf-a-3e35df-7f747b7fb-gdjfs\" (UID: \"97fbcbc0-ac3f-4957-a0f3-a6de47fb4a15\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3e35df-7f747b7fb-gdjfs"
Oct 30 19:40:28.512497 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.512049    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a47d8311-eef7-47b2-b426-140198d456ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3e35df-7874fb7595-2v6xk\" (UID: \"a47d8311-eef7-47b2-b426-140198d456ee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3e35df-7874fb7595-2v6xk"
Oct 30 19:40:28.622417 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.622151    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a47d8311-eef7-47b2-b426-140198d456ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3e35df-7874fb7595-2v6xk\" (UID: \"a47d8311-eef7-47b2-b426-140198d456ee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3e35df-7874fb7595-2v6xk"
Oct 30 19:40:28.622417 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.622248    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtzw5\" (UniqueName: \"kubernetes.io/projected/a47d8311-eef7-47b2-b426-140198d456ee-kube-api-access-jtzw5\") pod \"payload-pre-processing-e2e-inf-a-3e35df-7874fb7595-2v6xk\" (UID: \"a47d8311-eef7-47b2-b426-140198d456ee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3e35df-7874fb7595-2v6xk"
Oct 30 19:40:28.622417 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.622321    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97fbcbc0-ac3f-4957-a0f3-a6de47fb4a15-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3e35df-7f747b7fb-gdjfs\" (UID: \"97fbcbc0-ac3f-4957-a0f3-a6de47fb4a15\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3e35df-7f747b7fb-gdjfs"
Oct 30 19:40:28.622417 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.622348    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z9kv\" (UniqueName: \"kubernetes.io/projected/97fbcbc0-ac3f-4957-a0f3-a6de47fb4a15-kube-api-access-2z9kv\") pod \"payload-processing-e2e-inf-a-3e35df-7f747b7fb-gdjfs\" (UID: \"97fbcbc0-ac3f-4957-a0f3-a6de47fb4a15\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3e35df-7f747b7fb-gdjfs"
Oct 30 19:40:28.624250 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.624094    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a47d8311-eef7-47b2-b426-140198d456ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3e35df-7874fb7595-2v6xk\" (UID: \"a47d8311-eef7-47b2-b426-140198d456ee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3e35df-7874fb7595-2v6xk"
Oct 30 19:40:28.626297 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.624539    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97fbcbc0-ac3f-4957-a0f3-a6de47fb4a15-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3e35df-7f747b7fb-gdjfs\" (UID: \"97fbcbc0-ac3f-4957-a0f3-a6de47fb4a15\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3e35df-7f747b7fb-gdjfs"
Oct 30 19:40:28.638515 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.637660    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtzw5\" (UniqueName: \"kubernetes.io/projected/a47d8311-eef7-47b2-b426-140198d456ee-kube-api-access-jtzw5\") pod \"payload-pre-processing-e2e-inf-a-3e35df-7874fb7595-2v6xk\" (UID: \"a47d8311-eef7-47b2-b426-140198d456ee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3e35df-7874fb7595-2v6xk"
Oct 30 19:40:28.644311 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.644261    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z9kv\" (UniqueName: \"kubernetes.io/projected/97fbcbc0-ac3f-4957-a0f3-a6de47fb4a15-kube-api-access-2z9kv\") pod \"payload-processing-e2e-inf-a-3e35df-7f747b7fb-gdjfs\" (UID: \"97fbcbc0-ac3f-4957-a0f3-a6de47fb4a15\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3e35df-7f747b7fb-gdjfs"
Oct 30 19:40:28.655069 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.654871    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2db29833-fd7c-4dfd-88ba-6ae264d9f8b5" path="/var/lib/kubelet/pods/2db29833-fd7c-4dfd-88ba-6ae264d9f8b5/volumes"
Oct 30 19:40:28.704793 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.702259    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3e35df-7f747b7fb-gdjfs"
Oct 30 19:40:28.764398 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.764015    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp"
Oct 30 19:40:28.858391 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.858112    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3e35df-7874fb7595-2v6xk"
Oct 30 19:40:28.929856 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.929822    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a74dccf1-3de7-4f70-a549-281361992615-kube-api-access-sphlb\" (UniqueName: \"kubernetes.io/projected/a74dccf1-3de7-4f70-a549-281361992615-kube-api-access-sphlb\") pod \"a74dccf1-3de7-4f70-a549-281361992615\" (UID: \"a74dccf1-3de7-4f70-a549-281361992615\") "
Oct 30 19:40:28.930010 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.929911    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a74dccf1-3de7-4f70-a549-281361992615-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a74dccf1-3de7-4f70-a549-281361992615-istio-envoy\") pod \"a74dccf1-3de7-4f70-a549-281361992615\" (UID: \"a74dccf1-3de7-4f70-a549-281361992615\") "
Oct 30 19:40:28.930010 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.929997    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a74dccf1-3de7-4f70-a549-281361992615-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a74dccf1-3de7-4f70-a549-281361992615-istiod-ca-cert\") pod \"a74dccf1-3de7-4f70-a549-281361992615\" (UID: \"a74dccf1-3de7-4f70-a549-281361992615\") "
Oct 30 19:40:28.930214 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.930171    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a74dccf1-3de7-4f70-a549-281361992615-istio-token\" (UniqueName: \"kubernetes.io/projected/a74dccf1-3de7-4f70-a549-281361992615-istio-token\") pod \"a74dccf1-3de7-4f70-a549-281361992615\" (UID: \"a74dccf1-3de7-4f70-a549-281361992615\") "
Oct 30 19:40:28.930497 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.930452    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a74dccf1-3de7-4f70-a549-281361992615-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74dccf1-3de7-4f70-a549-281361992615-workload-socket\") pod \"a74dccf1-3de7-4f70-a549-281361992615\" (UID: \"a74dccf1-3de7-4f70-a549-281361992615\") "
Oct 30 19:40:28.930602 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.930543    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a74dccf1-3de7-4f70-a549-281361992615-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74dccf1-3de7-4f70-a549-281361992615-credential-socket\") pod \"a74dccf1-3de7-4f70-a549-281361992615\" (UID: \"a74dccf1-3de7-4f70-a549-281361992615\") "
Oct 30 19:40:28.930602 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.930572    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a74dccf1-3de7-4f70-a549-281361992615-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a74dccf1-3de7-4f70-a549-281361992615-workload-certs\") pod \"a74dccf1-3de7-4f70-a549-281361992615\" (UID: \"a74dccf1-3de7-4f70-a549-281361992615\") "
Oct 30 19:40:28.930772 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.930602    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a74dccf1-3de7-4f70-a549-281361992615-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a74dccf1-3de7-4f70-a549-281361992615-istio-podinfo\") pod \"a74dccf1-3de7-4f70-a549-281361992615\" (UID: \"a74dccf1-3de7-4f70-a549-281361992615\") "
Oct 30 19:40:28.930842 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.930771    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a74dccf1-3de7-4f70-a549-281361992615-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a74dccf1-3de7-4f70-a549-281361992615-istio-data\") pod \"a74dccf1-3de7-4f70-a549-281361992615\" (UID: \"a74dccf1-3de7-4f70-a549-281361992615\") "
Oct 30 19:40:28.932283 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.931325    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74dccf1-3de7-4f70-a549-281361992615-workload-socket" pod "a74dccf1-3de7-4f70-a549-281361992615" (UID: "a74dccf1-3de7-4f70-a549-281361992615"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:28.932283 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.931549    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74dccf1-3de7-4f70-a549-281361992615-workload-certs" pod "a74dccf1-3de7-4f70-a549-281361992615" (UID: "a74dccf1-3de7-4f70-a549-281361992615"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:28.932283 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.931849    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74dccf1-3de7-4f70-a549-281361992615-credential-socket" pod "a74dccf1-3de7-4f70-a549-281361992615" (UID: "a74dccf1-3de7-4f70-a549-281361992615"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:28.932283 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.931881    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74dccf1-3de7-4f70-a549-281361992615-istio-data" pod "a74dccf1-3de7-4f70-a549-281361992615" (UID: "a74dccf1-3de7-4f70-a549-281361992615"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:28.932283 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.931996    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a74dccf1-3de7-4f70-a549-281361992615-istiod-ca-cert" pod "a74dccf1-3de7-4f70-a549-281361992615" (UID: "a74dccf1-3de7-4f70-a549-281361992615"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:40:28.932283 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.932240    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74dccf1-3de7-4f70-a549-281361992615-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:28.932283 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.932258    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a74dccf1-3de7-4f70-a549-281361992615-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:28.941235 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.940987    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a74dccf1-3de7-4f70-a549-281361992615-istio-token" pod "a74dccf1-3de7-4f70-a549-281361992615" (UID: "a74dccf1-3de7-4f70-a549-281361992615"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:40:28.941235 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.941094    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74dccf1-3de7-4f70-a549-281361992615-istio-envoy" pod "a74dccf1-3de7-4f70-a549-281361992615" (UID: "a74dccf1-3de7-4f70-a549-281361992615"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:28.941235 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.941207    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a74dccf1-3de7-4f70-a549-281361992615-kube-api-access-sphlb" pod "a74dccf1-3de7-4f70-a549-281361992615" (UID: "a74dccf1-3de7-4f70-a549-281361992615"). InnerVolumeSpecName "kube-api-access-sphlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:40:28.942347 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:28.942299    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a74dccf1-3de7-4f70-a549-281361992615-istio-podinfo" pod "a74dccf1-3de7-4f70-a549-281361992615" (UID: "a74dccf1-3de7-4f70-a549-281361992615"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:40:29.037857 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:29.037701    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sphlb\" (UniqueName: \"kubernetes.io/projected/a74dccf1-3de7-4f70-a549-281361992615-kube-api-access-sphlb\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:29.037857 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:29.037756    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a74dccf1-3de7-4f70-a549-281361992615-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:29.037857 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:29.037773    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a74dccf1-3de7-4f70-a549-281361992615-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:29.037857 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:29.037786    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a74dccf1-3de7-4f70-a549-281361992615-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:29.037857 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:29.037802    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74dccf1-3de7-4f70-a549-281361992615-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:29.037857 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:29.037817    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a74dccf1-3de7-4f70-a549-281361992615-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:29.037857 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:29.037831    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a74dccf1-3de7-4f70-a549-281361992615-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:29.088598 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:29.088560    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3e35df-7f747b7fb-gdjfs"]
Oct 30 19:40:29.123875 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:29.121113    5043 generic.go:356] "Generic (PLEG): container finished" podID="a74dccf1-3de7-4f70-a549-281361992615" containerID="a9ba0d5574dfb6107ce27928d476f6e1716cc4ae0765f3742efbf5c3ce01a001" exitCode=0
Oct 30 19:40:29.123875 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:29.121297    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp"
Oct 30 19:40:29.146501 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:29.144890    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp" event={"ID":"a74dccf1-3de7-4f70-a549-281361992615","Type":"ContainerDied","Data":"a9ba0d5574dfb6107ce27928d476f6e1716cc4ae0765f3742efbf5c3ce01a001"}
Oct 30 19:40:29.146501 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:29.145020    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp" event={"ID":"a74dccf1-3de7-4f70-a549-281361992615","Type":"ContainerDied","Data":"9c40a602b3adc37a56963dd0f958fc71ee33d3ca829e427a5d9ebd661b8cbe3f"}
Oct 30 19:40:29.146501 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:29.145066    5043 scope.go:122] "RemoveContainer" containerID="a9ba0d5574dfb6107ce27928d476f6e1716cc4ae0765f3742efbf5c3ce01a001"
Oct 30 19:40:29.211245 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:29.210450    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3e35df-7874fb7595-2v6xk"]
Oct 30 19:40:29.218190 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:29.217560    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp"]
Oct 30 19:40:29.225867 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:29.225806    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-d1d256-openshift-default-979b4ccbf-lm6hp"]
Oct 30 19:40:29.226726 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:40:29.226695    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda47d8311_eef7_47b2_b426_140198d456ee.slice/crio-28faac7eb52b795139e82858cfd36e6551360655c2cc26b6d6ca12e739ed106e WatchSource:0}: Error finding container 28faac7eb52b795139e82858cfd36e6551360655c2cc26b6d6ca12e739ed106e: Status 404 returned error can't find the container with id 28faac7eb52b795139e82858cfd36e6551360655c2cc26b6d6ca12e739ed106e
Oct 30 19:40:29.233986 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:29.233969    5043 scope.go:122] "RemoveContainer" containerID="27ac769c108e2a9c3268d15f324d53c35afa80c74599451c9d15b3ea126d1e9e"
Oct 30 19:40:29.275906 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:29.275870    5043 patch_prober.go:29] interesting pod/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:40:29.276072 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:29.275932    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r" podUID="76b4e8e9-54ac-4461-b99c-a6f0361505b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:40:29.278527 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:29.278412    5043 scope.go:122] "RemoveContainer" containerID="a9ba0d5574dfb6107ce27928d476f6e1716cc4ae0765f3742efbf5c3ce01a001"
Oct 30 19:40:29.279004 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:40:29.278924    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9ba0d5574dfb6107ce27928d476f6e1716cc4ae0765f3742efbf5c3ce01a001\": container with ID starting with a9ba0d5574dfb6107ce27928d476f6e1716cc4ae0765f3742efbf5c3ce01a001 not found: ID does not exist" containerID="a9ba0d5574dfb6107ce27928d476f6e1716cc4ae0765f3742efbf5c3ce01a001"
Oct 30 19:40:29.279004 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:29.278961    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9ba0d5574dfb6107ce27928d476f6e1716cc4ae0765f3742efbf5c3ce01a001"} err="failed to get container status \"a9ba0d5574dfb6107ce27928d476f6e1716cc4ae0765f3742efbf5c3ce01a001\": rpc error: code = NotFound desc = could not find container \"a9ba0d5574dfb6107ce27928d476f6e1716cc4ae0765f3742efbf5c3ce01a001\": container with ID starting with a9ba0d5574dfb6107ce27928d476f6e1716cc4ae0765f3742efbf5c3ce01a001 not found: ID does not exist"
Oct 30 19:40:29.279004 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:29.278984    5043 scope.go:122] "RemoveContainer" containerID="27ac769c108e2a9c3268d15f324d53c35afa80c74599451c9d15b3ea126d1e9e"
Oct 30 19:40:29.279636 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:40:29.279605    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27ac769c108e2a9c3268d15f324d53c35afa80c74599451c9d15b3ea126d1e9e\": container with ID starting with 27ac769c108e2a9c3268d15f324d53c35afa80c74599451c9d15b3ea126d1e9e not found: ID does not exist" containerID="27ac769c108e2a9c3268d15f324d53c35afa80c74599451c9d15b3ea126d1e9e"
Oct 30 19:40:29.279850 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:29.279809    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27ac769c108e2a9c3268d15f324d53c35afa80c74599451c9d15b3ea126d1e9e"} err="failed to get container status \"27ac769c108e2a9c3268d15f324d53c35afa80c74599451c9d15b3ea126d1e9e\": rpc error: code = NotFound desc = could not find container \"27ac769c108e2a9c3268d15f324d53c35afa80c74599451c9d15b3ea126d1e9e\": container with ID starting with 27ac769c108e2a9c3268d15f324d53c35afa80c74599451c9d15b3ea126d1e9e not found: ID does not exist"
Oct 30 19:40:30.209034 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:30.208994    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3e35df-7f747b7fb-gdjfs" event={"ID":"97fbcbc0-ac3f-4957-a0f3-a6de47fb4a15","Type":"ContainerStarted","Data":"10a34c67bbd27a947f82efd6f7fbb7e19dbc8e60f02b3fa2e4d566c834afbac4"}
Oct 30 19:40:30.444074 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:30.209052    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3e35df-7f747b7fb-gdjfs" event={"ID":"97fbcbc0-ac3f-4957-a0f3-a6de47fb4a15","Type":"ContainerStarted","Data":"24e90f3b75a8d74aca4d22b03943a913f7e30621f8db1b24f9e5bf7c1b2a245e"}
Oct 30 19:40:30.444074 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:30.209079    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3e35df-7f747b7fb-gdjfs"
Oct 30 19:40:30.444074 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:30.212416    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3e35df-7874fb7595-2v6xk" event={"ID":"a47d8311-eef7-47b2-b426-140198d456ee","Type":"ContainerStarted","Data":"55451d2e63dcffbe130ecce3215310f15e8f4545b6160d879fc1f0b99386b22e"}
Oct 30 19:40:30.444074 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:30.212452    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3e35df-7874fb7595-2v6xk" event={"ID":"a47d8311-eef7-47b2-b426-140198d456ee","Type":"ContainerStarted","Data":"28faac7eb52b795139e82858cfd36e6551360655c2cc26b6d6ca12e739ed106e"}
Oct 30 19:40:30.444074 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:30.212484    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3e35df-7874fb7595-2v6xk"
Oct 30 19:40:30.444074 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:30.231212    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-3e35df-7f747b7fb-gdjfs" podStartSLOduration=2.231200011 podStartE2EDuration="2.231200011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:30.226123661 +0000 UTC m=+1449.737183627" watchObservedRunningTime="2026-10-30 19:40:30.231200011 +0000 UTC m=+1449.742259976"
Oct 30 19:40:30.444074 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:30.241681    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3e35df-7874fb7595-2v6xk" podStartSLOduration=2.241669216 podStartE2EDuration="2.241669216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:30.241038099 +0000 UTC m=+1449.752098090" watchObservedRunningTime="2026-10-30 19:40:30.241669216 +0000 UTC m=+1449.752729181"
Oct 30 19:40:30.663777 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:30.660070    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a74dccf1-3de7-4f70-a549-281361992615" path="/var/lib/kubelet/pods/a74dccf1-3de7-4f70-a549-281361992615/volumes"
Oct 30 19:40:31.222828 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:31.222798    5043 generic.go:356] "Generic (PLEG): container finished" podID="76b4e8e9-54ac-4461-b99c-a6f0361505b6" containerID="17ea88734869b20fec43ff49cb62bc641cda682522f45e6b64cc406eaeb3daf5" exitCode=0
Oct 30 19:40:31.223308 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:31.222880    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r" event={"ID":"76b4e8e9-54ac-4461-b99c-a6f0361505b6","Type":"ContainerDied","Data":"17ea88734869b20fec43ff49cb62bc641cda682522f45e6b64cc406eaeb3daf5"}
Oct 30 19:40:31.223384 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:31.223339    5043 scope.go:122] "RemoveContainer" containerID="17ea88734869b20fec43ff49cb62bc641cda682522f45e6b64cc406eaeb3daf5"
Oct 30 19:40:31.224773 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:31.224745    5043 generic.go:356] "Generic (PLEG): container finished" podID="7c584699-3a9f-45e1-a380-cb724ff74820" containerID="0ca07795e91bd72eaa34df2f3bb17a19d451880cb6df0d83257b6302915182b7" exitCode=0
Oct 30 19:40:31.224888 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:31.224768    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb" event={"ID":"7c584699-3a9f-45e1-a380-cb724ff74820","Type":"ContainerDied","Data":"0ca07795e91bd72eaa34df2f3bb17a19d451880cb6df0d83257b6302915182b7"}
Oct 30 19:40:31.225215 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:31.225179    5043 scope.go:122] "RemoveContainer" containerID="0ca07795e91bd72eaa34df2f3bb17a19d451880cb6df0d83257b6302915182b7"
Oct 30 19:40:31.225440 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:31.225415    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:40:31.225518 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:31.225494    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:40:31.225585 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:31.225545    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:40:31.226801 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:31.226772    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:40:31.226889 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:31.226838    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:40:31.226889 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:31.226872    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:40:32.226763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:32.224666    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ac37c8/test-model-ac37c8-kserve-6f996db89c-xpd6c"]
Oct 30 19:40:32.254397 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:32.254353    5043 cpu_manager.go:414] "containerMap: removing container" podUID="a74dccf1-3de7-4f70-a549-281361992615" containerName="istio-proxy"
Oct 30 19:40:32.254660 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:32.254642    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a74dccf1-3de7-4f70-a549-281361992615" containerName="istio-proxy"
Oct 30 19:40:32.254875 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:32.254849    5043 cpu_manager.go:414] "containerMap: removing container" podUID="a74dccf1-3de7-4f70-a549-281361992615" containerName="istio-proxy"
Oct 30 19:40:32.255020 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:32.255005    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a74dccf1-3de7-4f70-a549-281361992615" containerName="istio-proxy"
Oct 30 19:40:32.255385 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:32.255355    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="a74dccf1-3de7-4f70-a549-281361992615" containerName="istio-proxy"
Oct 30 19:40:32.255529 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:32.255498    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="a74dccf1-3de7-4f70-a549-281361992615" containerName="istio-proxy"
Oct 30 19:40:32.321986 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:32.321948    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ac37c8/test-model-ac37c8-kserve-6f996db89c-xpd6c"]
Oct 30 19:40:32.322181 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:32.322087    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ac37c8/test-model-ac37c8-kserve-6f996db89c-xpd6c"
Oct 30 19:40:32.335335 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:32.327254    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ac37c8\"/\"default-dockercfg-mkm4d\""
Oct 30 19:40:32.335335 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:32.327423    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ac37c8\"/\"test-model-ac37c8-kserve-self-signed-certs\""
Oct 30 19:40:32.335335 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:32.327567    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ac37c8\"/\"kube-root-ca.crt\""
Oct 30 19:40:32.335335 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:32.327660    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ac37c8\"/\"openshift-service-ca.crt\""
Oct 30 19:40:32.342199 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:32.342166    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79d37674-5d73-4b92-a57f-66bee5d7574c-tmp-dir\") pod \"test-model-ac37c8-kserve-6f996db89c-xpd6c\" (UID: \"79d37674-5d73-4b92-a57f-66bee5d7574c\") " pod="ai-tenant-e2e-inf-b-ac37c8/test-model-ac37c8-kserve-6f996db89c-xpd6c"
Oct 30 19:40:32.342614 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:32.342586    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll7rc\" (UniqueName: \"kubernetes.io/projected/79d37674-5d73-4b92-a57f-66bee5d7574c-kube-api-access-ll7rc\") pod \"test-model-ac37c8-kserve-6f996db89c-xpd6c\" (UID: \"79d37674-5d73-4b92-a57f-66bee5d7574c\") " pod="ai-tenant-e2e-inf-b-ac37c8/test-model-ac37c8-kserve-6f996db89c-xpd6c"
Oct 30 19:40:32.343014 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:32.342973    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79d37674-5d73-4b92-a57f-66bee5d7574c-model-cache\") pod \"test-model-ac37c8-kserve-6f996db89c-xpd6c\" (UID: \"79d37674-5d73-4b92-a57f-66bee5d7574c\") " pod="ai-tenant-e2e-inf-b-ac37c8/test-model-ac37c8-kserve-6f996db89c-xpd6c"
Oct 30 19:40:32.343251 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:32.343230    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79d37674-5d73-4b92-a57f-66bee5d7574c-home\") pod \"test-model-ac37c8-kserve-6f996db89c-xpd6c\" (UID: \"79d37674-5d73-4b92-a57f-66bee5d7574c\") " pod="ai-tenant-e2e-inf-b-ac37c8/test-model-ac37c8-kserve-6f996db89c-xpd6c"
Oct 30 19:40:32.343557 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:32.343519    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79d37674-5d73-4b92-a57f-66bee5d7574c-dshm\") pod \"test-model-ac37c8-kserve-6f996db89c-xpd6c\" (UID: \"79d37674-5d73-4b92-a57f-66bee5d7574c\") " pod="ai-tenant-e2e-inf-b-ac37c8/test-model-ac37c8-kserve-6f996db89c-xpd6c"
Oct 30 19:40:32.353896 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:32.353870    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79d37674-5d73-4b92-a57f-66bee5d7574c-tls-certs\") pod \"test-model-ac37c8-kserve-6f996db89c-xpd6c\" (UID: \"79d37674-5d73-4b92-a57f-66bee5d7574c\") " pod="ai-tenant-e2e-inf-b-ac37c8/test-model-ac37c8-kserve-6f996db89c-xpd6c"
Oct 30 19:40:32.354390 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:32.348913    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r" event={"ID":"76b4e8e9-54ac-4461-b99c-a6f0361505b6","Type":"ContainerStarted","Data":"8308f4a0b3eb68229f823b999921178a37ea37806d32fd6f180b18d02137155e"}
Oct 30 19:40:32.354509 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:32.354430    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb" event={"ID":"7c584699-3a9f-45e1-a380-cb724ff74820","Type":"ContainerStarted","Data":"7b4c6cf472d14e180bf7d8576147d493600fb998f62737c8a2636f6ff955e151"}
Oct 30 19:40:32.618673 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:32.618571    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79d37674-5d73-4b92-a57f-66bee5d7574c-model-cache\") pod \"test-model-ac37c8-kserve-6f996db89c-xpd6c\" (UID: \"79d37674-5d73-4b92-a57f-66bee5d7574c\") " pod="ai-tenant-e2e-inf-b-ac37c8/test-model-ac37c8-kserve-6f996db89c-xpd6c"
Oct 30 19:40:32.619040 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:32.619017    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79d37674-5d73-4b92-a57f-66bee5d7574c-home\") pod \"test-model-ac37c8-kserve-6f996db89c-xpd6c\" (UID: \"79d37674-5d73-4b92-a57f-66bee5d7574c\") " pod="ai-tenant-e2e-inf-b-ac37c8/test-model-ac37c8-kserve-6f996db89c-xpd6c"
Oct 30 19:40:32.619279 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:32.619263    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79d37674-5d73-4b92-a57f-66bee5d7574c-dshm\") pod \"test-model-ac37c8-kserve-6f996db89c-xpd6c\" (UID: \"79d37674-5d73-4b92-a57f-66bee5d7574c\") " pod="ai-tenant-e2e-inf-b-ac37c8/test-model-ac37c8-kserve-6f996db89c-xpd6c"
Oct 30 19:40:32.625764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:32.619460    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79d37674-5d73-4b92-a57f-66bee5d7574c-tls-certs\") pod \"test-model-ac37c8-kserve-6f996db89c-xpd6c\" (UID: \"79d37674-5d73-4b92-a57f-66bee5d7574c\") " pod="ai-tenant-e2e-inf-b-ac37c8/test-model-ac37c8-kserve-6f996db89c-xpd6c"
Oct 30 19:40:32.626049 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:32.626025    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79d37674-5d73-4b92-a57f-66bee5d7574c-tmp-dir\") pod \"test-model-ac37c8-kserve-6f996db89c-xpd6c\" (UID: \"79d37674-5d73-4b92-a57f-66bee5d7574c\") " pod="ai-tenant-e2e-inf-b-ac37c8/test-model-ac37c8-kserve-6f996db89c-xpd6c"
Oct 30 19:40:32.626342 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:32.626325    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll7rc\" (UniqueName: \"kubernetes.io/projected/79d37674-5d73-4b92-a57f-66bee5d7574c-kube-api-access-ll7rc\") pod \"test-model-ac37c8-kserve-6f996db89c-xpd6c\" (UID: \"79d37674-5d73-4b92-a57f-66bee5d7574c\") " pod="ai-tenant-e2e-inf-b-ac37c8/test-model-ac37c8-kserve-6f996db89c-xpd6c"
Oct 30 19:40:32.654278 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:32.627395    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79d37674-5d73-4b92-a57f-66bee5d7574c-model-cache\") pod \"test-model-ac37c8-kserve-6f996db89c-xpd6c\" (UID: \"79d37674-5d73-4b92-a57f-66bee5d7574c\") " pod="ai-tenant-e2e-inf-b-ac37c8/test-model-ac37c8-kserve-6f996db89c-xpd6c"
Oct 30 19:40:32.720187 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:32.643310    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79d37674-5d73-4b92-a57f-66bee5d7574c-home\") pod \"test-model-ac37c8-kserve-6f996db89c-xpd6c\" (UID: \"79d37674-5d73-4b92-a57f-66bee5d7574c\") " pod="ai-tenant-e2e-inf-b-ac37c8/test-model-ac37c8-kserve-6f996db89c-xpd6c"
Oct 30 19:40:32.720459 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:32.644244    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll7rc\" (UniqueName: \"kubernetes.io/projected/79d37674-5d73-4b92-a57f-66bee5d7574c-kube-api-access-ll7rc\") pod \"test-model-ac37c8-kserve-6f996db89c-xpd6c\" (UID: \"79d37674-5d73-4b92-a57f-66bee5d7574c\") " pod="ai-tenant-e2e-inf-b-ac37c8/test-model-ac37c8-kserve-6f996db89c-xpd6c"
Oct 30 19:40:32.720585 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:32.644481    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79d37674-5d73-4b92-a57f-66bee5d7574c-tmp-dir\") pod \"test-model-ac37c8-kserve-6f996db89c-xpd6c\" (UID: \"79d37674-5d73-4b92-a57f-66bee5d7574c\") " pod="ai-tenant-e2e-inf-b-ac37c8/test-model-ac37c8-kserve-6f996db89c-xpd6c"
Oct 30 19:40:32.720700 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:32.646879    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79d37674-5d73-4b92-a57f-66bee5d7574c-dshm\") pod \"test-model-ac37c8-kserve-6f996db89c-xpd6c\" (UID: \"79d37674-5d73-4b92-a57f-66bee5d7574c\") " pod="ai-tenant-e2e-inf-b-ac37c8/test-model-ac37c8-kserve-6f996db89c-xpd6c"
Oct 30 19:40:32.720847 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:32.649101    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79d37674-5d73-4b92-a57f-66bee5d7574c-tls-certs\") pod \"test-model-ac37c8-kserve-6f996db89c-xpd6c\" (UID: \"79d37674-5d73-4b92-a57f-66bee5d7574c\") " pod="ai-tenant-e2e-inf-b-ac37c8/test-model-ac37c8-kserve-6f996db89c-xpd6c"
Oct 30 19:40:32.976763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:32.965041    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ac37c8/test-model-ac37c8-kserve-6f996db89c-xpd6c"
Oct 30 19:40:33.226629 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:33.226195    5043 patch_prober.go:29] interesting pod/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body=
Oct 30 19:40:33.226629 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:33.226259    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r" podUID="76b4e8e9-54ac-4461-b99c-a6f0361505b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused"
Oct 30 19:40:33.226629 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:33.226559    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r"
Oct 30 19:40:33.355130 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:33.355097    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb"
Oct 30 19:40:33.355336 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:33.355208    5043 patch_prober.go:29] interesting pod/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Oct 30 19:40:33.355336 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:33.355248    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb" podUID="7c584699-3a9f-45e1-a380-cb724ff74820" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Oct 30 19:40:33.732950 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:33.729342    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ac37c8/test-model-ac37c8-kserve-6f996db89c-xpd6c"]
Oct 30 19:40:33.732950 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:40:33.731637    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79d37674_5d73_4b92_a57f_66bee5d7574c.slice/crio-fd577e752e56a6dd17e418eb407a332bff08662d060e7c32d0222577344b157f WatchSource:0}: Error finding container fd577e752e56a6dd17e418eb407a332bff08662d060e7c32d0222577344b157f: Status 404 returned error can't find the container with id fd577e752e56a6dd17e418eb407a332bff08662d060e7c32d0222577344b157f
Oct 30 19:40:34.049723 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:34.049686    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-22af66-7b5b988746-mm9zl"]
Oct 30 19:40:34.054425 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:34.054152    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-22af66-5bf7f46cd7-ffx8p"]
Oct 30 19:40:34.243807 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:34.243701    5043 patch_prober.go:29] interesting pod/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body=
Oct 30 19:40:34.244385 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:34.244349    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r" podUID="76b4e8e9-54ac-4461-b99c-a6f0361505b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused"
Oct 30 19:40:34.570870 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:34.570300    5043 patch_prober.go:29] interesting pod/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Oct 30 19:40:34.571471 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:34.571291    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb" podUID="7c584699-3a9f-45e1-a380-cb724ff74820" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Oct 30 19:40:34.721803 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:34.716595    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ac37c8/test-model-ac37c8-kserve-6f996db89c-xpd6c" event={"ID":"79d37674-5d73-4b92-a57f-66bee5d7574c","Type":"ContainerStarted","Data":"3381e9be570960018305c822463ee7a9d58d291982c99948ca87a22753ae60ba"}
Oct 30 19:40:34.721803 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:34.716656    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ac37c8/test-model-ac37c8-kserve-6f996db89c-xpd6c" event={"ID":"79d37674-5d73-4b92-a57f-66bee5d7574c","Type":"ContainerStarted","Data":"fd577e752e56a6dd17e418eb407a332bff08662d060e7c32d0222577344b157f"}
Oct 30 19:40:34.721803 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:34.719777    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ac37c8/test-model-ac37c8-kserve-6f996db89c-xpd6c" podStartSLOduration=2.719759241 podStartE2EDuration="2.719759241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:34.697160164 +0000 UTC m=+1454.208220130" watchObservedRunningTime="2026-10-30 19:40:34.719759241 +0000 UTC m=+1454.230819209"
Oct 30 19:40:34.966439 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:34.966393    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ac37c8/test-model-ac37c8-kserve-6f996db89c-xpd6c"
Oct 30 19:40:35.223013 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:35.222552    5043 patch_prober.go:29] interesting pod/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body=
Oct 30 19:40:35.223013 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:35.222610    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r" podUID="76b4e8e9-54ac-4461-b99c-a6f0361505b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused"
Oct 30 19:40:35.363763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:35.355783    5043 patch_prober.go:29] interesting pod/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Oct 30 19:40:35.363763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:35.355847    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb" podUID="7c584699-3a9f-45e1-a380-cb724ff74820" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Oct 30 19:40:36.223754 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:36.222759    5043 patch_prober.go:29] interesting pod/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body=
Oct 30 19:40:36.223754 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:36.222813    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r" podUID="76b4e8e9-54ac-4461-b99c-a6f0361505b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused"
Oct 30 19:40:36.362764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:36.362122    5043 patch_prober.go:29] interesting pod/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Oct 30 19:40:36.362764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:36.362182    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb" podUID="7c584699-3a9f-45e1-a380-cb724ff74820" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Oct 30 19:40:37.045210 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:37.045151    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74211168-bplxw"]
Oct 30 19:40:37.075762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:37.073005    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74211168-bplxw"]
Oct 30 19:40:37.222929 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:37.222893    5043 patch_prober.go:29] interesting pod/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body=
Oct 30 19:40:37.223115 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:37.222965    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r" podUID="76b4e8e9-54ac-4461-b99c-a6f0361505b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused"
Oct 30 19:40:37.355198 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:37.355125    5043 patch_prober.go:29] interesting pod/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Oct 30 19:40:37.355418 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:37.355397    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb" podUID="7c584699-3a9f-45e1-a380-cb724ff74820" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Oct 30 19:40:38.300688 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:38.300570    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r"
Oct 30 19:40:38.301217 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:38.301168    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r"
Oct 30 19:40:38.302498 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:38.302475    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r"
Oct 30 19:40:38.360910 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:38.360871    5043 patch_prober.go:29] interesting pod/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Oct 30 19:40:38.361103 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:38.360931    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb" podUID="7c584699-3a9f-45e1-a380-cb724ff74820" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Oct 30 19:40:38.648999 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:38.648907    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd788ac5-1205-49d2-8ed4-8c22703a79ec" path="/var/lib/kubelet/pods/fd788ac5-1205-49d2-8ed4-8c22703a79ec/volumes"
Oct 30 19:40:38.742141 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:38.742106    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb"
Oct 30 19:40:39.051245 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:39.051205    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-22af66-7b5b988746-mm9zl" podUID="183c301b-de2c-4f03-b142-d53819964452" containerName="payload-processing" containerID="cri-o://3337b9dfef89f070dd0a168af76f962595f790f202464508cb16ee6dbfd653be" gracePeriod=25
Oct 30 19:40:39.054783 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:39.054712    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-22af66-5bf7f46cd7-ffx8p" podUID="8c12af95-b3fd-4146-857b-27c42aa4dc74" containerName="payload-pre-processing" containerID="cri-o://54ca03c314cafb968d3574a7afda707d1baa18147ee1390af40ac827affc31d0" gracePeriod=25
Oct 30 19:40:39.394865 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:39.394834    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-22af66-5bf7f46cd7-ffx8p"
Oct 30 19:40:39.430517 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:39.429869    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c12af95-b3fd-4146-857b-27c42aa4dc74-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c12af95-b3fd-4146-857b-27c42aa4dc74-plugins-config-volume\") pod \"8c12af95-b3fd-4146-857b-27c42aa4dc74\" (UID: \"8c12af95-b3fd-4146-857b-27c42aa4dc74\") "
Oct 30 19:40:39.430517 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:39.430059    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c12af95-b3fd-4146-857b-27c42aa4dc74-kube-api-access-grsjj\" (UniqueName: \"kubernetes.io/projected/8c12af95-b3fd-4146-857b-27c42aa4dc74-kube-api-access-grsjj\") pod \"8c12af95-b3fd-4146-857b-27c42aa4dc74\" (UID: \"8c12af95-b3fd-4146-857b-27c42aa4dc74\") "
Oct 30 19:40:39.433325 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:39.433293    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb"
Oct 30 19:40:39.434125 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:39.434093    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c12af95-b3fd-4146-857b-27c42aa4dc74-plugins-config-volume" pod "8c12af95-b3fd-4146-857b-27c42aa4dc74" (UID: "8c12af95-b3fd-4146-857b-27c42aa4dc74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:40:39.440672 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:39.440635    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c12af95-b3fd-4146-857b-27c42aa4dc74-kube-api-access-grsjj" pod "8c12af95-b3fd-4146-857b-27c42aa4dc74" (UID: "8c12af95-b3fd-4146-857b-27c42aa4dc74"). InnerVolumeSpecName "kube-api-access-grsjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:40:39.470815 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:39.470729    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-22af66-7b5b988746-mm9zl"
Oct 30 19:40:39.533760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:39.533704    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/183c301b-de2c-4f03-b142-d53819964452-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/183c301b-de2c-4f03-b142-d53819964452-plugins-config-volume\") pod \"183c301b-de2c-4f03-b142-d53819964452\" (UID: \"183c301b-de2c-4f03-b142-d53819964452\") "
Oct 30 19:40:39.534022 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:39.534005    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/183c301b-de2c-4f03-b142-d53819964452-kube-api-access-gkd7z\" (UniqueName: \"kubernetes.io/projected/183c301b-de2c-4f03-b142-d53819964452-kube-api-access-gkd7z\") pod \"183c301b-de2c-4f03-b142-d53819964452\" (UID: \"183c301b-de2c-4f03-b142-d53819964452\") "
Oct 30 19:40:39.534248 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:39.534216    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/183c301b-de2c-4f03-b142-d53819964452-plugins-config-volume" pod "183c301b-de2c-4f03-b142-d53819964452" (UID: "183c301b-de2c-4f03-b142-d53819964452"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:40:39.534638 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:39.534618    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grsjj\" (UniqueName: \"kubernetes.io/projected/8c12af95-b3fd-4146-857b-27c42aa4dc74-kube-api-access-grsjj\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:39.534721 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:39.534645    5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/183c301b-de2c-4f03-b142-d53819964452-plugins-config-volume\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:39.534721 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:39.534659    5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c12af95-b3fd-4146-857b-27c42aa4dc74-plugins-config-volume\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:39.537238 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:39.537211    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/183c301b-de2c-4f03-b142-d53819964452-kube-api-access-gkd7z" pod "183c301b-de2c-4f03-b142-d53819964452" (UID: "183c301b-de2c-4f03-b142-d53819964452"). InnerVolumeSpecName "kube-api-access-gkd7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:40:39.636867 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:39.636780    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkd7z\" (UniqueName: \"kubernetes.io/projected/183c301b-de2c-4f03-b142-d53819964452-kube-api-access-gkd7z\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:39.742781 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:39.742715    5043 generic.go:356] "Generic (PLEG): container finished" podID="183c301b-de2c-4f03-b142-d53819964452" containerID="3337b9dfef89f070dd0a168af76f962595f790f202464508cb16ee6dbfd653be" exitCode=0
Oct 30 19:40:39.742781 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:39.742768    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-22af66-7b5b988746-mm9zl" event={"ID":"183c301b-de2c-4f03-b142-d53819964452","Type":"ContainerDied","Data":"3337b9dfef89f070dd0a168af76f962595f790f202464508cb16ee6dbfd653be"}
Oct 30 19:40:39.742781 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:39.742796    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-22af66-7b5b988746-mm9zl"
Oct 30 19:40:39.743110 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:39.742843    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-22af66-7b5b988746-mm9zl" event={"ID":"183c301b-de2c-4f03-b142-d53819964452","Type":"ContainerDied","Data":"a0627b5aefbc93810c4eefd6756791b974437f70d550f4b900a239182ef4a967"}
Oct 30 19:40:39.743110 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:39.742872    5043 scope.go:122] "RemoveContainer" containerID="3337b9dfef89f070dd0a168af76f962595f790f202464508cb16ee6dbfd653be"
Oct 30 19:40:39.744787 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:39.744762    5043 generic.go:356] "Generic (PLEG): container finished" podID="8c12af95-b3fd-4146-857b-27c42aa4dc74" containerID="54ca03c314cafb968d3574a7afda707d1baa18147ee1390af40ac827affc31d0" exitCode=0
Oct 30 19:40:39.745016 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:39.744833    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-22af66-5bf7f46cd7-ffx8p"
Oct 30 19:40:39.745105 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:39.744849    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-22af66-5bf7f46cd7-ffx8p" event={"ID":"8c12af95-b3fd-4146-857b-27c42aa4dc74","Type":"ContainerDied","Data":"54ca03c314cafb968d3574a7afda707d1baa18147ee1390af40ac827affc31d0"}
Oct 30 19:40:39.745171 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:39.745123    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-22af66-5bf7f46cd7-ffx8p" event={"ID":"8c12af95-b3fd-4146-857b-27c42aa4dc74","Type":"ContainerDied","Data":"19cb30b379fd8815c00a221cdc4473214d174fe0ceace125107d5a7f9e992521"}
Oct 30 19:40:39.746585 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:39.746567    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb"
Oct 30 19:40:39.757899 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:39.757868    5043 scope.go:122] "RemoveContainer" containerID="3337b9dfef89f070dd0a168af76f962595f790f202464508cb16ee6dbfd653be"
Oct 30 19:40:39.758208 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:40:39.758172    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3337b9dfef89f070dd0a168af76f962595f790f202464508cb16ee6dbfd653be\": container with ID starting with 3337b9dfef89f070dd0a168af76f962595f790f202464508cb16ee6dbfd653be not found: ID does not exist" containerID="3337b9dfef89f070dd0a168af76f962595f790f202464508cb16ee6dbfd653be"
Oct 30 19:40:39.758281 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:39.758209    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3337b9dfef89f070dd0a168af76f962595f790f202464508cb16ee6dbfd653be"} err="failed to get container status \"3337b9dfef89f070dd0a168af76f962595f790f202464508cb16ee6dbfd653be\": rpc error: code = NotFound desc = could not find container \"3337b9dfef89f070dd0a168af76f962595f790f202464508cb16ee6dbfd653be\": container with ID starting with 3337b9dfef89f070dd0a168af76f962595f790f202464508cb16ee6dbfd653be not found: ID does not exist"
Oct 30 19:40:39.758281 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:39.758232    5043 scope.go:122] "RemoveContainer" containerID="54ca03c314cafb968d3574a7afda707d1baa18147ee1390af40ac827affc31d0"
Oct 30 19:40:39.770973 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:39.770952    5043 scope.go:122] "RemoveContainer" containerID="54ca03c314cafb968d3574a7afda707d1baa18147ee1390af40ac827affc31d0"
Oct 30 19:40:39.771325 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:40:39.771298    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54ca03c314cafb968d3574a7afda707d1baa18147ee1390af40ac827affc31d0\": container with ID starting with 54ca03c314cafb968d3574a7afda707d1baa18147ee1390af40ac827affc31d0 not found: ID does not exist" containerID="54ca03c314cafb968d3574a7afda707d1baa18147ee1390af40ac827affc31d0"
Oct 30 19:40:39.771415 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:39.771335    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54ca03c314cafb968d3574a7afda707d1baa18147ee1390af40ac827affc31d0"} err="failed to get container status \"54ca03c314cafb968d3574a7afda707d1baa18147ee1390af40ac827affc31d0\": rpc error: code = NotFound desc = could not find container \"54ca03c314cafb968d3574a7afda707d1baa18147ee1390af40ac827affc31d0\": container with ID starting with 54ca03c314cafb968d3574a7afda707d1baa18147ee1390af40ac827affc31d0 not found: ID does not exist"
Oct 30 19:40:39.793625 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:39.793572    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-22af66-7b5b988746-mm9zl"]
Oct 30 19:40:39.816763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:39.809561    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-22af66-7b5b988746-mm9zl"]
Oct 30 19:40:39.819475 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:39.817156    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-22af66-5bf7f46cd7-ffx8p"]
Oct 30 19:40:39.822700 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:39.821405    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-22af66-5bf7f46cd7-ffx8p"]
Oct 30 19:40:40.652010 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:40.651981    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="183c301b-de2c-4f03-b142-d53819964452" path="/var/lib/kubelet/pods/183c301b-de2c-4f03-b142-d53819964452/volumes"
Oct 30 19:40:40.652513 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:40.652497    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c12af95-b3fd-4146-857b-27c42aa4dc74" path="/var/lib/kubelet/pods/8c12af95-b3fd-4146-857b-27c42aa4dc74/volumes"
Oct 30 19:40:41.229532 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:41.229501    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3e35df-7874fb7595-2v6xk"
Oct 30 19:40:41.229787 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:41.229584    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3e35df-7f747b7fb-gdjfs"
Oct 30 19:40:42.965571 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:42.965527    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ac37c8/test-model-ac37c8-kserve-6f996db89c-xpd6c"
Oct 30 19:40:42.981512 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:42.981484    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ac37c8/test-model-ac37c8-kserve-6f996db89c-xpd6c"
Oct 30 19:40:43.887461 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:43.887432    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ac37c8/test-model-ac37c8-kserve-6f996db89c-xpd6c"
Oct 30 19:40:54.981988 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:40:54.981947    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:40:57.919858 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:57.919822    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46"]
Oct 30 19:40:57.921612 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:57.921582    5043 cpu_manager.go:414] "containerMap: removing container" podUID="8c12af95-b3fd-4146-857b-27c42aa4dc74" containerName="payload-pre-processing"
Oct 30 19:40:57.921809 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:57.921794    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c12af95-b3fd-4146-857b-27c42aa4dc74" containerName="payload-pre-processing"
Oct 30 19:40:57.921944 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:57.921932    5043 cpu_manager.go:414] "containerMap: removing container" podUID="183c301b-de2c-4f03-b142-d53819964452" containerName="payload-processing"
Oct 30 19:40:57.922028 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:57.922017    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="183c301b-de2c-4f03-b142-d53819964452" containerName="payload-processing"
Oct 30 19:40:57.922343 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:57.922325    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c12af95-b3fd-4146-857b-27c42aa4dc74" containerName="payload-pre-processing"
Oct 30 19:40:57.922458 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:57.922447    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="183c301b-de2c-4f03-b142-d53819964452" containerName="payload-processing"
Oct 30 19:40:57.928668 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:57.928631    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46"]
Oct 30 19:40:57.928888 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:57.928770    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46"
Oct 30 19:40:57.933181 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:57.933151    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-47110f-openshift-default-dockercfg-vqr44\""
Oct 30 19:40:57.972364 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:57.972320    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-istio-token\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46\" (UID: \"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46"
Oct 30 19:40:57.972570 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:57.972386    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-credential-socket\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46\" (UID: \"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46"
Oct 30 19:40:57.972570 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:57.972491    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-istio-envoy\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46\" (UID: \"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46"
Oct 30 19:40:57.972710 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:57.972575    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntnlc\" (UniqueName: \"kubernetes.io/projected/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-kube-api-access-ntnlc\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46\" (UID: \"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46"
Oct 30 19:40:57.972805 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:57.972727    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-workload-certs\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46\" (UID: \"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46"
Oct 30 19:40:57.972864 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:57.972841    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-istio-data\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46\" (UID: \"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46"
Oct 30 19:40:57.972954 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:57.972884    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-istiod-ca-cert\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46\" (UID: \"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46"
Oct 30 19:40:57.972954 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:57.972928    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-istio-podinfo\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46\" (UID: \"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46"
Oct 30 19:40:57.973204 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:57.973159    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-workload-socket\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46\" (UID: \"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46"
Oct 30 19:40:58.081759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:58.079828    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-workload-certs\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46\" (UID: \"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46"
Oct 30 19:40:58.081759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:58.079916    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-istio-data\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46\" (UID: \"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46"
Oct 30 19:40:58.081759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:58.079962    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-istiod-ca-cert\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46\" (UID: \"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46"
Oct 30 19:40:58.081759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:58.080006    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-istio-podinfo\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46\" (UID: \"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46"
Oct 30 19:40:58.081759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:58.080086    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-workload-socket\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46\" (UID: \"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46"
Oct 30 19:40:58.081759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:58.080148    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-istio-token\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46\" (UID: \"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46"
Oct 30 19:40:58.081759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:58.080185    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-credential-socket\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46\" (UID: \"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46"
Oct 30 19:40:58.081759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:58.080216    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-istio-envoy\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46\" (UID: \"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46"
Oct 30 19:40:58.081759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:58.080305    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntnlc\" (UniqueName: \"kubernetes.io/projected/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-kube-api-access-ntnlc\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46\" (UID: \"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46"
Oct 30 19:40:58.081759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:58.081092    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-workload-certs\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46\" (UID: \"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46"
Oct 30 19:40:58.081759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:58.081318    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-istio-data\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46\" (UID: \"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46"
Oct 30 19:40:58.082410 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:58.081931    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-istiod-ca-cert\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46\" (UID: \"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46"
Oct 30 19:40:58.102035 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:58.083164    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-workload-socket\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46\" (UID: \"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46"
Oct 30 19:40:58.102035 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:58.083457    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-credential-socket\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46\" (UID: \"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46"
Oct 30 19:40:58.172828 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:58.164420    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntnlc\" (UniqueName: \"kubernetes.io/projected/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-kube-api-access-ntnlc\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46\" (UID: \"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46"
Oct 30 19:40:58.172828 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:58.164661    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-istio-envoy\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46\" (UID: \"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46"
Oct 30 19:40:58.190759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:58.183114    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-istio-podinfo\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46\" (UID: \"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46"
Oct 30 19:40:58.190759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:58.188358    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-istio-token\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46\" (UID: \"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46"
Oct 30 19:40:58.249679 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:58.249608    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46"
Oct 30 19:40:58.630944 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:40:58.630535    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd2e9ea0_1574_4cd8_9e79_206d8f06dc8e.slice/crio-eb265abed4fa4a38e43190bd6c7bc60442559f1d3d691caffe26a5f834a1a5be WatchSource:0}: Error finding container eb265abed4fa4a38e43190bd6c7bc60442559f1d3d691caffe26a5f834a1a5be: Status 404 returned error can't find the container with id eb265abed4fa4a38e43190bd6c7bc60442559f1d3d691caffe26a5f834a1a5be
Oct 30 19:40:58.632091 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:58.632063    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46"]
Oct 30 19:40:58.634561 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:58.634520    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:40:58.634688 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:58.634617    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:40:58.634688 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:58.634668    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:40:59.178945 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:59.178912    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46" event={"ID":"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e","Type":"ContainerStarted","Data":"6c16073da015e4edf494cce7836e9ace33fde259c9c9db2a491e02bdbfce47f1"}
Oct 30 19:40:59.179466 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:59.178969    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46" event={"ID":"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e","Type":"ContainerStarted","Data":"eb265abed4fa4a38e43190bd6c7bc60442559f1d3d691caffe26a5f834a1a5be"}
Oct 30 19:40:59.242175 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:59.242122    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46" podStartSLOduration=2.242107673 podStartE2EDuration="2.242107673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:59.239172829 +0000 UTC m=+1478.750232795" watchObservedRunningTime="2026-10-30 19:40:59.242107673 +0000 UTC m=+1478.753167638"
Oct 30 19:40:59.253797 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:59.253769    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46"
Oct 30 19:40:59.259954 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:40:59.259926    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46"
Oct 30 19:41:00.191940 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:00.191849    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46"
Oct 30 19:41:00.192886 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:00.192863    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46"
Oct 30 19:41:02.935701 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:02.935668    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-47110f-65f4f5b6b4-5528c"]
Oct 30 19:41:02.940793 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:02.940730    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-47110f-65f4f5b6b4-5528c"
Oct 30 19:41:02.942627 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:02.942603    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-47110f-7cdc4b47f5-2brw4"]
Oct 30 19:41:02.946989 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:02.946958    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-47110f-7cdc4b47f5-2brw4"
Oct 30 19:41:02.951574 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:02.951549    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-47110f-dockercfg-zqg96\""
Oct 30 19:41:02.952265 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:02.952236    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-47110f\""
Oct 30 19:41:02.953333 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:02.953313    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-47110f-65f4f5b6b4-5528c"]
Oct 30 19:41:03.042999 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:02.962108    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-47110f-7cdc4b47f5-2brw4"]
Oct 30 19:41:03.100801 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:03.100766    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83c175fe-cdfd-4a54-88b6-f9a2061a11dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-47110f-7cdc4b47f5-2brw4\" (UID: \"83c175fe-cdfd-4a54-88b6-f9a2061a11dd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-47110f-7cdc4b47f5-2brw4"
Oct 30 19:41:03.101240 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:03.101215    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6fz9\" (UniqueName: \"kubernetes.io/projected/83c175fe-cdfd-4a54-88b6-f9a2061a11dd-kube-api-access-c6fz9\") pod \"payload-pre-processing-e2e-inf-b-47110f-7cdc4b47f5-2brw4\" (UID: \"83c175fe-cdfd-4a54-88b6-f9a2061a11dd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-47110f-7cdc4b47f5-2brw4"
Oct 30 19:41:03.101508 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:03.101481    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceac8507-3941-431a-9880-2fb98e79e257-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-47110f-65f4f5b6b4-5528c\" (UID: \"ceac8507-3941-431a-9880-2fb98e79e257\") " pod="openshift-ingress/payload-processing-e2e-inf-b-47110f-65f4f5b6b4-5528c"
Oct 30 19:41:03.101864 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:03.101843    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52tgp\" (UniqueName: \"kubernetes.io/projected/ceac8507-3941-431a-9880-2fb98e79e257-kube-api-access-52tgp\") pod \"payload-processing-e2e-inf-b-47110f-65f4f5b6b4-5528c\" (UID: \"ceac8507-3941-431a-9880-2fb98e79e257\") " pod="openshift-ingress/payload-processing-e2e-inf-b-47110f-65f4f5b6b4-5528c"
Oct 30 19:41:03.217485 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:03.216628    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52tgp\" (UniqueName: \"kubernetes.io/projected/ceac8507-3941-431a-9880-2fb98e79e257-kube-api-access-52tgp\") pod \"payload-processing-e2e-inf-b-47110f-65f4f5b6b4-5528c\" (UID: \"ceac8507-3941-431a-9880-2fb98e79e257\") " pod="openshift-ingress/payload-processing-e2e-inf-b-47110f-65f4f5b6b4-5528c"
Oct 30 19:41:03.246796 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:03.243279    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83c175fe-cdfd-4a54-88b6-f9a2061a11dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-47110f-7cdc4b47f5-2brw4\" (UID: \"83c175fe-cdfd-4a54-88b6-f9a2061a11dd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-47110f-7cdc4b47f5-2brw4"
Oct 30 19:41:03.246796 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:03.246692    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83c175fe-cdfd-4a54-88b6-f9a2061a11dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-47110f-7cdc4b47f5-2brw4\" (UID: \"83c175fe-cdfd-4a54-88b6-f9a2061a11dd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-47110f-7cdc4b47f5-2brw4"
Oct 30 19:41:03.247109 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:03.246847    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6fz9\" (UniqueName: \"kubernetes.io/projected/83c175fe-cdfd-4a54-88b6-f9a2061a11dd-kube-api-access-c6fz9\") pod \"payload-pre-processing-e2e-inf-b-47110f-7cdc4b47f5-2brw4\" (UID: \"83c175fe-cdfd-4a54-88b6-f9a2061a11dd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-47110f-7cdc4b47f5-2brw4"
Oct 30 19:41:03.247109 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:03.246909    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceac8507-3941-431a-9880-2fb98e79e257-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-47110f-65f4f5b6b4-5528c\" (UID: \"ceac8507-3941-431a-9880-2fb98e79e257\") " pod="openshift-ingress/payload-processing-e2e-inf-b-47110f-65f4f5b6b4-5528c"
Oct 30 19:41:03.263231 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:03.247662    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceac8507-3941-431a-9880-2fb98e79e257-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-47110f-65f4f5b6b4-5528c\" (UID: \"ceac8507-3941-431a-9880-2fb98e79e257\") " pod="openshift-ingress/payload-processing-e2e-inf-b-47110f-65f4f5b6b4-5528c"
Oct 30 19:41:03.263231 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:03.253039    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52tgp\" (UniqueName: \"kubernetes.io/projected/ceac8507-3941-431a-9880-2fb98e79e257-kube-api-access-52tgp\") pod \"payload-processing-e2e-inf-b-47110f-65f4f5b6b4-5528c\" (UID: \"ceac8507-3941-431a-9880-2fb98e79e257\") " pod="openshift-ingress/payload-processing-e2e-inf-b-47110f-65f4f5b6b4-5528c"
Oct 30 19:41:03.273161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:03.264320    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6fz9\" (UniqueName: \"kubernetes.io/projected/83c175fe-cdfd-4a54-88b6-f9a2061a11dd-kube-api-access-c6fz9\") pod \"payload-pre-processing-e2e-inf-b-47110f-7cdc4b47f5-2brw4\" (UID: \"83c175fe-cdfd-4a54-88b6-f9a2061a11dd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-47110f-7cdc4b47f5-2brw4"
Oct 30 19:41:03.273161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:03.264996    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-47110f-65f4f5b6b4-5528c"
Oct 30 19:41:03.312609 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:03.312525    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-47110f-7cdc4b47f5-2brw4"
Oct 30 19:41:03.583161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:03.583134    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-47110f-65f4f5b6b4-5528c"]
Oct 30 19:41:03.584022 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:41:03.583998    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceac8507_3941_431a_9880_2fb98e79e257.slice/crio-200a1167da944ab0a4e2931a3e4fe1ecb1b29cc20f275529453eb8efd2a00d88 WatchSource:0}: Error finding container 200a1167da944ab0a4e2931a3e4fe1ecb1b29cc20f275529453eb8efd2a00d88: Status 404 returned error can't find the container with id 200a1167da944ab0a4e2931a3e4fe1ecb1b29cc20f275529453eb8efd2a00d88
Oct 30 19:41:03.624400 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:03.624373    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-47110f-7cdc4b47f5-2brw4"]
Oct 30 19:41:03.624515 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:41:03.624415    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83c175fe_cdfd_4a54_88b6_f9a2061a11dd.slice/crio-bdc71f7a89cc77a81ae152c1b3a878324f0176c3a076d4d13030e9c2abbd1f2c WatchSource:0}: Error finding container bdc71f7a89cc77a81ae152c1b3a878324f0176c3a076d4d13030e9c2abbd1f2c: Status 404 returned error can't find the container with id bdc71f7a89cc77a81ae152c1b3a878324f0176c3a076d4d13030e9c2abbd1f2c
Oct 30 19:41:04.256515 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:04.256111    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-47110f-7cdc4b47f5-2brw4" event={"ID":"83c175fe-cdfd-4a54-88b6-f9a2061a11dd","Type":"ContainerStarted","Data":"6f7549fab0ce5ce26f5fd1a06edaaacc6b005037e60cac8a11744b52ceea452f"}
Oct 30 19:41:04.256991 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:04.256560    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-47110f-7cdc4b47f5-2brw4"
Oct 30 19:41:04.256991 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:04.256588    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-47110f-7cdc4b47f5-2brw4" event={"ID":"83c175fe-cdfd-4a54-88b6-f9a2061a11dd","Type":"ContainerStarted","Data":"bdc71f7a89cc77a81ae152c1b3a878324f0176c3a076d4d13030e9c2abbd1f2c"}
Oct 30 19:41:04.258356 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:04.258322    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-47110f-65f4f5b6b4-5528c" event={"ID":"ceac8507-3941-431a-9880-2fb98e79e257","Type":"ContainerStarted","Data":"235892c7afed574dbfdf735db599ba73f5cace485be26b1172bd1bc281fa23e0"}
Oct 30 19:41:04.258492 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:04.258381    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-47110f-65f4f5b6b4-5528c" event={"ID":"ceac8507-3941-431a-9880-2fb98e79e257","Type":"ContainerStarted","Data":"200a1167da944ab0a4e2931a3e4fe1ecb1b29cc20f275529453eb8efd2a00d88"}
Oct 30 19:41:04.258492 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:04.258453    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-47110f-65f4f5b6b4-5528c"
Oct 30 19:41:04.283631 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:04.283567    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-47110f-7cdc4b47f5-2brw4" podStartSLOduration=2.283548129 podStartE2EDuration="2.283548129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:04.277838389 +0000 UTC m=+1483.788898355" watchObservedRunningTime="2026-10-30 19:41:04.283548129 +0000 UTC m=+1483.794608095"
Oct 30 19:41:04.308781 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:04.308690    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-47110f-65f4f5b6b4-5528c" podStartSLOduration=2.308676206 podStartE2EDuration="2.308676206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:04.308144608 +0000 UTC m=+1483.819204574" watchObservedRunningTime="2026-10-30 19:41:04.308676206 +0000 UTC m=+1483.819736170"
Oct 30 19:41:09.398153 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:09.398116    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-22af66-5klt4"]
Oct 30 19:41:09.407598 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:09.407571    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-22af66-5klt4"]
Oct 30 19:41:10.648671 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:10.648634    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7b8ba26-6bb8-4666-b83f-808dc079edef" path="/var/lib/kubelet/pods/a7b8ba26-6bb8-4666-b83f-808dc079edef/volumes"
Oct 30 19:41:15.286294 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:15.286260    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-47110f-65f4f5b6b4-5528c"
Oct 30 19:41:15.286813 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:15.286702    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-47110f-7cdc4b47f5-2brw4"
Oct 30 19:41:19.202273 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:19.202228    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-59f017-p2l7c"]
Oct 30 19:41:19.208125 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:19.208107    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-59f017-p2l7c"
Oct 30 19:41:19.213777 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:19.213756    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:41:19.213876 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:19.213833    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:41:19.213876 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:19.213850    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:41:19.217343 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:19.217323    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:41:19.224267 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:19.224244    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-59f017-p2l7c"]
Oct 30 19:41:19.283411 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:19.283361    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc220459-8640-4c7f-b2a9-55b67d3a9a70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-59f017-p2l7c\" (UID: \"fc220459-8640-4c7f-b2a9-55b67d3a9a70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-59f017-p2l7c"
Oct 30 19:41:19.283761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:19.283724    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc220459-8640-4c7f-b2a9-55b67d3a9a70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-59f017-p2l7c\" (UID: \"fc220459-8640-4c7f-b2a9-55b67d3a9a70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-59f017-p2l7c"
Oct 30 19:41:19.283949 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:19.283911    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dsvg\" (UniqueName: \"kubernetes.io/projected/fc220459-8640-4c7f-b2a9-55b67d3a9a70-kube-api-access-8dsvg\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-59f017-p2l7c\" (UID: \"fc220459-8640-4c7f-b2a9-55b67d3a9a70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-59f017-p2l7c"
Oct 30 19:41:19.385997 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:19.385959    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc220459-8640-4c7f-b2a9-55b67d3a9a70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-59f017-p2l7c\" (UID: \"fc220459-8640-4c7f-b2a9-55b67d3a9a70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-59f017-p2l7c"
Oct 30 19:41:19.386196 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:19.386047    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc220459-8640-4c7f-b2a9-55b67d3a9a70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-59f017-p2l7c\" (UID: \"fc220459-8640-4c7f-b2a9-55b67d3a9a70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-59f017-p2l7c"
Oct 30 19:41:19.386196 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:19.386084    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dsvg\" (UniqueName: \"kubernetes.io/projected/fc220459-8640-4c7f-b2a9-55b67d3a9a70-kube-api-access-8dsvg\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-59f017-p2l7c\" (UID: \"fc220459-8640-4c7f-b2a9-55b67d3a9a70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-59f017-p2l7c"
Oct 30 19:41:19.386819 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:19.386795    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc220459-8640-4c7f-b2a9-55b67d3a9a70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-59f017-p2l7c\" (UID: \"fc220459-8640-4c7f-b2a9-55b67d3a9a70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-59f017-p2l7c"
Oct 30 19:41:19.388680 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:19.388653    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc220459-8640-4c7f-b2a9-55b67d3a9a70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-59f017-p2l7c\" (UID: \"fc220459-8640-4c7f-b2a9-55b67d3a9a70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-59f017-p2l7c"
Oct 30 19:41:19.404835 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:19.404802    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dsvg\" (UniqueName: \"kubernetes.io/projected/fc220459-8640-4c7f-b2a9-55b67d3a9a70-kube-api-access-8dsvg\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-59f017-p2l7c\" (UID: \"fc220459-8640-4c7f-b2a9-55b67d3a9a70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-59f017-p2l7c"
Oct 30 19:41:19.525156 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:19.525127    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-59f017-p2l7c"
Oct 30 19:41:19.837789 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:19.835293    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-59f017-p2l7c"]
Oct 30 19:41:20.046155 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:20.046112    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2356d951-gm2s9"]
Oct 30 19:41:20.058382 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:20.058345    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2356d951-gm2s9"]
Oct 30 19:41:20.474922 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:20.474883    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-59f017-p2l7c" event={"ID":"fc220459-8640-4c7f-b2a9-55b67d3a9a70","Type":"ContainerStarted","Data":"846ca16a269d995d1cdfeae2a29aab3f6bc1901261c8681c2d2b68b09ce76cfe"}
Oct 30 19:41:20.475430 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:20.474953    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-59f017-p2l7c" event={"ID":"fc220459-8640-4c7f-b2a9-55b67d3a9a70","Type":"ContainerStarted","Data":"29236f5905c536c47067d3b5e1a7605accdb3c6c964e98bbf7cb43af6c6e2fcf"}
Oct 30 19:41:20.494596 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:20.494538    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-59f017-p2l7c" podStartSLOduration=1.494521473 podStartE2EDuration="1.494521473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:20.492793584 +0000 UTC m=+1500.003853554" watchObservedRunningTime="2026-10-30 19:41:20.494521473 +0000 UTC m=+1500.005581823"
Oct 30 19:41:20.649209 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:20.649128    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51fb5873-a661-491d-8437-aa1c915f126e" path="/var/lib/kubelet/pods/51fb5873-a661-491d-8437-aa1c915f126e/volumes"
Oct 30 19:41:21.115508 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:21.115481    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cmw6w_76e21916-0744-4499-a0cb-64a7a44a8285/console-operator/2.log"
Oct 30 19:41:21.120377 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:21.120349    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cmw6w_76e21916-0744-4499-a0cb-64a7a44a8285/console-operator/2.log"
Oct 30 19:41:21.489459 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:21.489425    5043 generic.go:356] "Generic (PLEG): container finished" podID="fc220459-8640-4c7f-b2a9-55b67d3a9a70" containerID="846ca16a269d995d1cdfeae2a29aab3f6bc1901261c8681c2d2b68b09ce76cfe" exitCode=0
Oct 30 19:41:21.489908 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:21.489522    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-59f017-p2l7c" event={"ID":"fc220459-8640-4c7f-b2a9-55b67d3a9a70","Type":"ContainerDied","Data":"846ca16a269d995d1cdfeae2a29aab3f6bc1901261c8681c2d2b68b09ce76cfe"}
Oct 30 19:41:22.705318 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:22.705291    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-59f017-p2l7c"
Oct 30 19:41:22.757230 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:22.757135    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc220459-8640-4c7f-b2a9-55b67d3a9a70-kube-api-access-8dsvg\" (UniqueName: \"kubernetes.io/projected/fc220459-8640-4c7f-b2a9-55b67d3a9a70-kube-api-access-8dsvg\") pod \"fc220459-8640-4c7f-b2a9-55b67d3a9a70\" (UID: \"fc220459-8640-4c7f-b2a9-55b67d3a9a70\") "
Oct 30 19:41:22.757230 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:22.757197    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc220459-8640-4c7f-b2a9-55b67d3a9a70-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc220459-8640-4c7f-b2a9-55b67d3a9a70-maas-api-openshift-ca\") pod \"fc220459-8640-4c7f-b2a9-55b67d3a9a70\" (UID: \"fc220459-8640-4c7f-b2a9-55b67d3a9a70\") "
Oct 30 19:41:22.757511 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:22.757327    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc220459-8640-4c7f-b2a9-55b67d3a9a70-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc220459-8640-4c7f-b2a9-55b67d3a9a70-maas-api-xks-ca\") pod \"fc220459-8640-4c7f-b2a9-55b67d3a9a70\" (UID: \"fc220459-8640-4c7f-b2a9-55b67d3a9a70\") "
Oct 30 19:41:22.758654 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:22.758621    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc220459-8640-4c7f-b2a9-55b67d3a9a70-maas-api-openshift-ca" pod "fc220459-8640-4c7f-b2a9-55b67d3a9a70" (UID: "fc220459-8640-4c7f-b2a9-55b67d3a9a70"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:41:22.761421 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:22.761376    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc220459-8640-4c7f-b2a9-55b67d3a9a70-kube-api-access-8dsvg" pod "fc220459-8640-4c7f-b2a9-55b67d3a9a70" (UID: "fc220459-8640-4c7f-b2a9-55b67d3a9a70"). InnerVolumeSpecName "kube-api-access-8dsvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:41:22.761928 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:22.761883    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc220459-8640-4c7f-b2a9-55b67d3a9a70-maas-api-xks-ca" pod "fc220459-8640-4c7f-b2a9-55b67d3a9a70" (UID: "fc220459-8640-4c7f-b2a9-55b67d3a9a70"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:41:22.859291 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:22.859253    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dsvg\" (UniqueName: \"kubernetes.io/projected/fc220459-8640-4c7f-b2a9-55b67d3a9a70-kube-api-access-8dsvg\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:22.859291 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:22.859285    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc220459-8640-4c7f-b2a9-55b67d3a9a70-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:22.859291 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:22.859298    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc220459-8640-4c7f-b2a9-55b67d3a9a70-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:23.510960 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:23.510933    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-59f017-p2l7c"
Oct 30 19:41:23.510960 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:23.510953    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-59f017-p2l7c" event={"ID":"fc220459-8640-4c7f-b2a9-55b67d3a9a70","Type":"ContainerDied","Data":"29236f5905c536c47067d3b5e1a7605accdb3c6c964e98bbf7cb43af6c6e2fcf"}
Oct 30 19:41:23.511174 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:23.511006    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29236f5905c536c47067d3b5e1a7605accdb3c6c964e98bbf7cb43af6c6e2fcf"
Oct 30 19:41:24.340584 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:24.340542    5043 scope.go:122] "RemoveContainer" containerID="03f6dc2bdc8eaa5ea83434fe26a50d3c3a4f9ac864823302368fb9f61325aea1"
Oct 30 19:41:24.354312 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:24.354291    5043 scope.go:122] "RemoveContainer" containerID="2176ca0d283de4eb2ef7baaca8a15906367fa321a97f6798f79337bbb0ddf971"
Oct 30 19:41:24.980715 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:41:24.980679    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:41:29.466345 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:29.466312    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-59f017-694c87888d-lw9vk"]
Oct 30 19:41:34.466891 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:34.466840    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-59f017-694c87888d-lw9vk" podUID="2f47551b-3e96-4cde-a128-757ef1095525" containerName="payload-processing" containerID="cri-o://bab150187234c85dbc13601712566c77cd93d9b2ccdb0fd2ea53b0716e71bec9" gracePeriod=25
Oct 30 19:41:34.621873 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:34.621844    5043 generic.go:356] "Generic (PLEG): container finished" podID="2f47551b-3e96-4cde-a128-757ef1095525" containerID="bab150187234c85dbc13601712566c77cd93d9b2ccdb0fd2ea53b0716e71bec9" exitCode=0
Oct 30 19:41:34.622129 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:34.621926    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-59f017-694c87888d-lw9vk" event={"ID":"2f47551b-3e96-4cde-a128-757ef1095525","Type":"ContainerDied","Data":"bab150187234c85dbc13601712566c77cd93d9b2ccdb0fd2ea53b0716e71bec9"}
Oct 30 19:41:34.781723 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:34.781693    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-59f017-694c87888d-lw9vk"
Oct 30 19:41:34.897933 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:34.897898    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f47551b-3e96-4cde-a128-757ef1095525-kube-api-access-bmdxv\" (UniqueName: \"kubernetes.io/projected/2f47551b-3e96-4cde-a128-757ef1095525-kube-api-access-bmdxv\") pod \"2f47551b-3e96-4cde-a128-757ef1095525\" (UID: \"2f47551b-3e96-4cde-a128-757ef1095525\") "
Oct 30 19:41:34.898160 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:34.898019    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f47551b-3e96-4cde-a128-757ef1095525-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f47551b-3e96-4cde-a128-757ef1095525-plugins-config-volume\") pod \"2f47551b-3e96-4cde-a128-757ef1095525\" (UID: \"2f47551b-3e96-4cde-a128-757ef1095525\") "
Oct 30 19:41:34.898447 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:34.898422    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f47551b-3e96-4cde-a128-757ef1095525-plugins-config-volume" pod "2f47551b-3e96-4cde-a128-757ef1095525" (UID: "2f47551b-3e96-4cde-a128-757ef1095525"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:41:34.898712 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:34.898692    5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f47551b-3e96-4cde-a128-757ef1095525-plugins-config-volume\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:34.900831 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:34.900796    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f47551b-3e96-4cde-a128-757ef1095525-kube-api-access-bmdxv" pod "2f47551b-3e96-4cde-a128-757ef1095525" (UID: "2f47551b-3e96-4cde-a128-757ef1095525"). InnerVolumeSpecName "kube-api-access-bmdxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:41:35.000926 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:35.000830    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmdxv\" (UniqueName: \"kubernetes.io/projected/2f47551b-3e96-4cde-a128-757ef1095525-kube-api-access-bmdxv\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:35.632536 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:35.632506    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-59f017-694c87888d-lw9vk"
Oct 30 19:41:35.633009 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:35.632507    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-59f017-694c87888d-lw9vk" event={"ID":"2f47551b-3e96-4cde-a128-757ef1095525","Type":"ContainerDied","Data":"850e0d1049895fb9777a1e049972d9cb22f356a47da03c1ad3245c0a061276e9"}
Oct 30 19:41:35.633009 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:35.632639    5043 scope.go:122] "RemoveContainer" containerID="bab150187234c85dbc13601712566c77cd93d9b2ccdb0fd2ea53b0716e71bec9"
Oct 30 19:41:35.660996 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:35.660963    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-59f017-694c87888d-lw9vk"]
Oct 30 19:41:35.667959 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:35.667924    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-59f017-694c87888d-lw9vk"]
Oct 30 19:41:36.723766 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:36.715976    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f47551b-3e96-4cde-a128-757ef1095525" path="/var/lib/kubelet/pods/2f47551b-3e96-4cde-a128-757ef1095525/volumes"
Oct 30 19:41:36.808762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:36.808707    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-59f017-p2l7c"]
Oct 30 19:41:36.818915 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:36.818883    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-59f017-p2l7c"]
Oct 30 19:41:38.649537 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:38.649501    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc220459-8640-4c7f-b2a9-55b67d3a9a70" path="/var/lib/kubelet/pods/fc220459-8640-4c7f-b2a9-55b67d3a9a70/volumes"
Oct 30 19:41:43.441049 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:43.441001    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp"]
Oct 30 19:41:43.441585 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:43.441338    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp" podUID="2299b666-7ae6-452c-bde8-203a6de2e0e2" containerName="istio-proxy" containerID="cri-o://34617ed9fdfeed4050ca97494268475c4287df5faa072b235b44741cccd6871b" gracePeriod=30
Oct 30 19:41:46.194763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:46.115090    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc"]
Oct 30 19:41:46.204773 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:46.195595    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc" podUID="b34f07ce-6edf-4493-b349-ad7916bb93aa" containerName="istio-proxy" containerID="cri-o://d1b0d067fd0d220353deb56c12459de13b426a871771643c434535c21c1ae5ff" gracePeriod=30
Oct 30 19:41:46.588169 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:46.588123    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ca9d14-qpd57"]
Oct 30 19:41:46.589363 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:46.589324    5043 cpu_manager.go:414] "containerMap: removing container" podUID="2f47551b-3e96-4cde-a128-757ef1095525" containerName="payload-processing"
Oct 30 19:41:46.589363 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:46.589357    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f47551b-3e96-4cde-a128-757ef1095525" containerName="payload-processing"
Oct 30 19:41:46.589612 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:46.589386    5043 cpu_manager.go:414] "containerMap: removing container" podUID="fc220459-8640-4c7f-b2a9-55b67d3a9a70" containerName="revoke-keys"
Oct 30 19:41:46.589612 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:46.589394    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc220459-8640-4c7f-b2a9-55b67d3a9a70" containerName="revoke-keys"
Oct 30 19:41:46.589612 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:46.589558    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f47551b-3e96-4cde-a128-757ef1095525" containerName="payload-processing"
Oct 30 19:41:46.589612 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:46.589572    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc220459-8640-4c7f-b2a9-55b67d3a9a70" containerName="revoke-keys"
Oct 30 19:41:46.594185 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:46.594161    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ca9d14-qpd57"
Oct 30 19:41:46.599041 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:46.599014    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:41:46.599277 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:46.599255    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:41:46.602185 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:46.602160    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ca9d14-qpd57"]
Oct 30 19:41:46.652237 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:46.652192    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/362c9035-e63c-425b-9826-a2453cf0aa18-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ca9d14-qpd57\" (UID: \"362c9035-e63c-425b-9826-a2453cf0aa18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ca9d14-qpd57"
Oct 30 19:41:46.652453 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:46.652311    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65v2t\" (UniqueName: \"kubernetes.io/projected/362c9035-e63c-425b-9826-a2453cf0aa18-kube-api-access-65v2t\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ca9d14-qpd57\" (UID: \"362c9035-e63c-425b-9826-a2453cf0aa18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ca9d14-qpd57"
Oct 30 19:41:46.652597 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:46.652578    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/362c9035-e63c-425b-9826-a2453cf0aa18-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ca9d14-qpd57\" (UID: \"362c9035-e63c-425b-9826-a2453cf0aa18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ca9d14-qpd57"
Oct 30 19:41:46.676418 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:46.676378    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:41:46.676788 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:46.676770    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:41:46.754248 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:46.754205    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/362c9035-e63c-425b-9826-a2453cf0aa18-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ca9d14-qpd57\" (UID: \"362c9035-e63c-425b-9826-a2453cf0aa18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ca9d14-qpd57"
Oct 30 19:41:46.754439 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:46.754270    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65v2t\" (UniqueName: \"kubernetes.io/projected/362c9035-e63c-425b-9826-a2453cf0aa18-kube-api-access-65v2t\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ca9d14-qpd57\" (UID: \"362c9035-e63c-425b-9826-a2453cf0aa18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ca9d14-qpd57"
Oct 30 19:41:46.754536 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:46.754512    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/362c9035-e63c-425b-9826-a2453cf0aa18-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ca9d14-qpd57\" (UID: \"362c9035-e63c-425b-9826-a2453cf0aa18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ca9d14-qpd57"
Oct 30 19:41:46.755217 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:46.755196    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/362c9035-e63c-425b-9826-a2453cf0aa18-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ca9d14-qpd57\" (UID: \"362c9035-e63c-425b-9826-a2453cf0aa18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ca9d14-qpd57"
Oct 30 19:41:46.791077 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:46.791009    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/362c9035-e63c-425b-9826-a2453cf0aa18-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ca9d14-qpd57\" (UID: \"362c9035-e63c-425b-9826-a2453cf0aa18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ca9d14-qpd57"
Oct 30 19:41:46.795107 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:46.795054    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65v2t\" (UniqueName: \"kubernetes.io/projected/362c9035-e63c-425b-9826-a2453cf0aa18-kube-api-access-65v2t\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ca9d14-qpd57\" (UID: \"362c9035-e63c-425b-9826-a2453cf0aa18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ca9d14-qpd57"
Oct 30 19:41:46.913513 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:46.913413    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ca9d14-qpd57"
Oct 30 19:41:47.235130 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:41:47.235103    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod362c9035_e63c_425b_9826_a2453cf0aa18.slice/crio-28edb4f76cc7f4e91adecddeebb8063e4187896a89cd0bc0a8d1cd1bddeae067 WatchSource:0}: Error finding container 28edb4f76cc7f4e91adecddeebb8063e4187896a89cd0bc0a8d1cd1bddeae067: Status 404 returned error can't find the container with id 28edb4f76cc7f4e91adecddeebb8063e4187896a89cd0bc0a8d1cd1bddeae067
Oct 30 19:41:47.236241 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:47.236214    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ca9d14-qpd57"]
Oct 30 19:41:47.904195 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:47.904094    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ca9d14-qpd57" event={"ID":"362c9035-e63c-425b-9826-a2453cf0aa18","Type":"ContainerStarted","Data":"dd63b14f79ec2e3d0e33e3fc7f1d05fea61742ecd679eddae0a83894802b14cc"}
Oct 30 19:41:47.904195 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:47.904150    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ca9d14-qpd57" event={"ID":"362c9035-e63c-425b-9826-a2453cf0aa18","Type":"ContainerStarted","Data":"28edb4f76cc7f4e91adecddeebb8063e4187896a89cd0bc0a8d1cd1bddeae067"}
Oct 30 19:41:47.926577 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:47.926506    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ca9d14-qpd57" podStartSLOduration=1.926487158 podStartE2EDuration="1.926487158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:47.920450944 +0000 UTC m=+1527.431510909" watchObservedRunningTime="2026-10-30 19:41:47.926487158 +0000 UTC m=+1527.437547319"
Oct 30 19:41:48.894954 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:48.894803    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp"
Oct 30 19:41:48.931205 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:48.931171    5043 generic.go:356] "Generic (PLEG): container finished" podID="362c9035-e63c-425b-9826-a2453cf0aa18" containerID="dd63b14f79ec2e3d0e33e3fc7f1d05fea61742ecd679eddae0a83894802b14cc" exitCode=0
Oct 30 19:41:48.931396 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:48.931246    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ca9d14-qpd57" event={"ID":"362c9035-e63c-425b-9826-a2453cf0aa18","Type":"ContainerDied","Data":"dd63b14f79ec2e3d0e33e3fc7f1d05fea61742ecd679eddae0a83894802b14cc"}
Oct 30 19:41:48.934410 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:48.934381    5043 generic.go:356] "Generic (PLEG): container finished" podID="2299b666-7ae6-452c-bde8-203a6de2e0e2" containerID="34617ed9fdfeed4050ca97494268475c4287df5faa072b235b44741cccd6871b" exitCode=0
Oct 30 19:41:48.934575 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:48.934441    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp"
Oct 30 19:41:48.934575 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:48.934554    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp" event={"ID":"2299b666-7ae6-452c-bde8-203a6de2e0e2","Type":"ContainerDied","Data":"34617ed9fdfeed4050ca97494268475c4287df5faa072b235b44741cccd6871b"}
Oct 30 19:41:48.934704 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:48.934598    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp" event={"ID":"2299b666-7ae6-452c-bde8-203a6de2e0e2","Type":"ContainerDied","Data":"3db8aae4b836013528387e54206961a3d6b5ae0d0c505e8058c287fa9404acb8"}
Oct 30 19:41:48.934704 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:48.934627    5043 scope.go:122] "RemoveContainer" containerID="34617ed9fdfeed4050ca97494268475c4287df5faa072b235b44741cccd6871b"
Oct 30 19:41:48.937454 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:48.937432    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2299b666-7ae6-452c-bde8-203a6de2e0e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2299b666-7ae6-452c-bde8-203a6de2e0e2-workload-socket\") pod \"2299b666-7ae6-452c-bde8-203a6de2e0e2\" (UID: \"2299b666-7ae6-452c-bde8-203a6de2e0e2\") "
Oct 30 19:41:48.937582 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:48.937513    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2299b666-7ae6-452c-bde8-203a6de2e0e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2299b666-7ae6-452c-bde8-203a6de2e0e2-credential-socket\") pod \"2299b666-7ae6-452c-bde8-203a6de2e0e2\" (UID: \"2299b666-7ae6-452c-bde8-203a6de2e0e2\") "
Oct 30 19:41:48.937582 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:48.937554    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2299b666-7ae6-452c-bde8-203a6de2e0e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2299b666-7ae6-452c-bde8-203a6de2e0e2-istio-envoy\") pod \"2299b666-7ae6-452c-bde8-203a6de2e0e2\" (UID: \"2299b666-7ae6-452c-bde8-203a6de2e0e2\") "
Oct 30 19:41:48.937699 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:48.937649    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2299b666-7ae6-452c-bde8-203a6de2e0e2-kube-api-access-zb8d6\" (UniqueName: \"kubernetes.io/projected/2299b666-7ae6-452c-bde8-203a6de2e0e2-kube-api-access-zb8d6\") pod \"2299b666-7ae6-452c-bde8-203a6de2e0e2\" (UID: \"2299b666-7ae6-452c-bde8-203a6de2e0e2\") "
Oct 30 19:41:48.937699 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:48.937674    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2299b666-7ae6-452c-bde8-203a6de2e0e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2299b666-7ae6-452c-bde8-203a6de2e0e2-istio-podinfo\") pod \"2299b666-7ae6-452c-bde8-203a6de2e0e2\" (UID: \"2299b666-7ae6-452c-bde8-203a6de2e0e2\") "
Oct 30 19:41:48.937699 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:48.937692    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2299b666-7ae6-452c-bde8-203a6de2e0e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2299b666-7ae6-452c-bde8-203a6de2e0e2-istiod-ca-cert\") pod \"2299b666-7ae6-452c-bde8-203a6de2e0e2\" (UID: \"2299b666-7ae6-452c-bde8-203a6de2e0e2\") "
Oct 30 19:41:48.937929 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:48.937715    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2299b666-7ae6-452c-bde8-203a6de2e0e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2299b666-7ae6-452c-bde8-203a6de2e0e2-istio-data\") pod \"2299b666-7ae6-452c-bde8-203a6de2e0e2\" (UID: \"2299b666-7ae6-452c-bde8-203a6de2e0e2\") "
Oct 30 19:41:48.938153 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:48.938124    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2299b666-7ae6-452c-bde8-203a6de2e0e2-workload-socket" pod "2299b666-7ae6-452c-bde8-203a6de2e0e2" (UID: "2299b666-7ae6-452c-bde8-203a6de2e0e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:48.938283 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:48.938243    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2299b666-7ae6-452c-bde8-203a6de2e0e2-istio-token\" (UniqueName: \"kubernetes.io/projected/2299b666-7ae6-452c-bde8-203a6de2e0e2-istio-token\") pod \"2299b666-7ae6-452c-bde8-203a6de2e0e2\" (UID: \"2299b666-7ae6-452c-bde8-203a6de2e0e2\") "
Oct 30 19:41:48.938512 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:48.938493    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2299b666-7ae6-452c-bde8-203a6de2e0e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2299b666-7ae6-452c-bde8-203a6de2e0e2-workload-certs\") pod \"2299b666-7ae6-452c-bde8-203a6de2e0e2\" (UID: \"2299b666-7ae6-452c-bde8-203a6de2e0e2\") "
Oct 30 19:41:48.938675 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:48.938649    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2299b666-7ae6-452c-bde8-203a6de2e0e2-istiod-ca-cert" pod "2299b666-7ae6-452c-bde8-203a6de2e0e2" (UID: "2299b666-7ae6-452c-bde8-203a6de2e0e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:41:48.939120 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:48.939101    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2299b666-7ae6-452c-bde8-203a6de2e0e2-credential-socket" pod "2299b666-7ae6-452c-bde8-203a6de2e0e2" (UID: "2299b666-7ae6-452c-bde8-203a6de2e0e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:48.939453 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:48.939427    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2299b666-7ae6-452c-bde8-203a6de2e0e2-workload-certs" pod "2299b666-7ae6-452c-bde8-203a6de2e0e2" (UID: "2299b666-7ae6-452c-bde8-203a6de2e0e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:48.939712 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:48.939684    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2299b666-7ae6-452c-bde8-203a6de2e0e2-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:48.939850 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:48.939831    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2299b666-7ae6-452c-bde8-203a6de2e0e2-istio-data" pod "2299b666-7ae6-452c-bde8-203a6de2e0e2" (UID: "2299b666-7ae6-452c-bde8-203a6de2e0e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:48.939927 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:48.939837    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2299b666-7ae6-452c-bde8-203a6de2e0e2-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:48.939927 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:48.939872    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2299b666-7ae6-452c-bde8-203a6de2e0e2-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:48.939927 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:48.939885    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2299b666-7ae6-452c-bde8-203a6de2e0e2-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:48.940934 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:48.940907    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2299b666-7ae6-452c-bde8-203a6de2e0e2-istio-envoy" pod "2299b666-7ae6-452c-bde8-203a6de2e0e2" (UID: "2299b666-7ae6-452c-bde8-203a6de2e0e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:48.941032 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:48.940929    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2299b666-7ae6-452c-bde8-203a6de2e0e2-kube-api-access-zb8d6" pod "2299b666-7ae6-452c-bde8-203a6de2e0e2" (UID: "2299b666-7ae6-452c-bde8-203a6de2e0e2"). InnerVolumeSpecName "kube-api-access-zb8d6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:41:48.941087 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:48.941050    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2299b666-7ae6-452c-bde8-203a6de2e0e2-istio-token" pod "2299b666-7ae6-452c-bde8-203a6de2e0e2" (UID: "2299b666-7ae6-452c-bde8-203a6de2e0e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:41:48.941255 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:48.941235    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2299b666-7ae6-452c-bde8-203a6de2e0e2-istio-podinfo" pod "2299b666-7ae6-452c-bde8-203a6de2e0e2" (UID: "2299b666-7ae6-452c-bde8-203a6de2e0e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:41:48.999036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:48.998881    5043 scope.go:122] "RemoveContainer" containerID="42320e18a73d22e0f6d2ecea24d2f4c60977246febcac7def4b7fe01f4b077a4"
Oct 30 19:41:49.044413 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:49.044371    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2299b666-7ae6-452c-bde8-203a6de2e0e2-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:49.044413 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:49.044411    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2299b666-7ae6-452c-bde8-203a6de2e0e2-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:49.044677 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:49.044427    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb8d6\" (UniqueName: \"kubernetes.io/projected/2299b666-7ae6-452c-bde8-203a6de2e0e2-kube-api-access-zb8d6\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:49.044677 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:49.044439    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2299b666-7ae6-452c-bde8-203a6de2e0e2-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:49.044677 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:49.044452    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2299b666-7ae6-452c-bde8-203a6de2e0e2-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:49.054903 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:49.054879    5043 scope.go:122] "RemoveContainer" containerID="34617ed9fdfeed4050ca97494268475c4287df5faa072b235b44741cccd6871b"
Oct 30 19:41:49.055259 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:41:49.055234    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34617ed9fdfeed4050ca97494268475c4287df5faa072b235b44741cccd6871b\": container with ID starting with 34617ed9fdfeed4050ca97494268475c4287df5faa072b235b44741cccd6871b not found: ID does not exist" containerID="34617ed9fdfeed4050ca97494268475c4287df5faa072b235b44741cccd6871b"
Oct 30 19:41:49.055362 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:49.055272    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34617ed9fdfeed4050ca97494268475c4287df5faa072b235b44741cccd6871b"} err="failed to get container status \"34617ed9fdfeed4050ca97494268475c4287df5faa072b235b44741cccd6871b\": rpc error: code = NotFound desc = could not find container \"34617ed9fdfeed4050ca97494268475c4287df5faa072b235b44741cccd6871b\": container with ID starting with 34617ed9fdfeed4050ca97494268475c4287df5faa072b235b44741cccd6871b not found: ID does not exist"
Oct 30 19:41:49.055362 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:49.055303    5043 scope.go:122] "RemoveContainer" containerID="42320e18a73d22e0f6d2ecea24d2f4c60977246febcac7def4b7fe01f4b077a4"
Oct 30 19:41:49.055531 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:41:49.055507    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42320e18a73d22e0f6d2ecea24d2f4c60977246febcac7def4b7fe01f4b077a4\": container with ID starting with 42320e18a73d22e0f6d2ecea24d2f4c60977246febcac7def4b7fe01f4b077a4 not found: ID does not exist" containerID="42320e18a73d22e0f6d2ecea24d2f4c60977246febcac7def4b7fe01f4b077a4"
Oct 30 19:41:49.055605 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:49.055536    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42320e18a73d22e0f6d2ecea24d2f4c60977246febcac7def4b7fe01f4b077a4"} err="failed to get container status \"42320e18a73d22e0f6d2ecea24d2f4c60977246febcac7def4b7fe01f4b077a4\": rpc error: code = NotFound desc = could not find container \"42320e18a73d22e0f6d2ecea24d2f4c60977246febcac7def4b7fe01f4b077a4\": container with ID starting with 42320e18a73d22e0f6d2ecea24d2f4c60977246febcac7def4b7fe01f4b077a4 not found: ID does not exist"
Oct 30 19:41:49.335908 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:49.335828    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp"]
Oct 30 19:41:49.341534 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:49.341478    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-d2xbp"]
Oct 30 19:41:50.238271 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:50.238243    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ca9d14-qpd57"
Oct 30 19:41:50.393243 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:50.393198    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/362c9035-e63c-425b-9826-a2453cf0aa18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/362c9035-e63c-425b-9826-a2453cf0aa18-maas-api-openshift-ca\") pod \"362c9035-e63c-425b-9826-a2453cf0aa18\" (UID: \"362c9035-e63c-425b-9826-a2453cf0aa18\") "
Oct 30 19:41:50.393468 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:50.393265    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/362c9035-e63c-425b-9826-a2453cf0aa18-kube-api-access-65v2t\" (UniqueName: \"kubernetes.io/projected/362c9035-e63c-425b-9826-a2453cf0aa18-kube-api-access-65v2t\") pod \"362c9035-e63c-425b-9826-a2453cf0aa18\" (UID: \"362c9035-e63c-425b-9826-a2453cf0aa18\") "
Oct 30 19:41:50.393590 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:50.393572    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/362c9035-e63c-425b-9826-a2453cf0aa18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/362c9035-e63c-425b-9826-a2453cf0aa18-maas-api-xks-ca\") pod \"362c9035-e63c-425b-9826-a2453cf0aa18\" (UID: \"362c9035-e63c-425b-9826-a2453cf0aa18\") "
Oct 30 19:41:50.394459 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:50.394430    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/362c9035-e63c-425b-9826-a2453cf0aa18-maas-api-openshift-ca" pod "362c9035-e63c-425b-9826-a2453cf0aa18" (UID: "362c9035-e63c-425b-9826-a2453cf0aa18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:41:50.396916 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:50.396851    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/362c9035-e63c-425b-9826-a2453cf0aa18-maas-api-xks-ca" pod "362c9035-e63c-425b-9826-a2453cf0aa18" (UID: "362c9035-e63c-425b-9826-a2453cf0aa18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:41:50.397020 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:50.396913    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/362c9035-e63c-425b-9826-a2453cf0aa18-kube-api-access-65v2t" pod "362c9035-e63c-425b-9826-a2453cf0aa18" (UID: "362c9035-e63c-425b-9826-a2453cf0aa18"). InnerVolumeSpecName "kube-api-access-65v2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:41:50.495949 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:50.495917    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/362c9035-e63c-425b-9826-a2453cf0aa18-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:50.496168 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:50.496155    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/362c9035-e63c-425b-9826-a2453cf0aa18-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:50.496233 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:50.496225    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65v2t\" (UniqueName: \"kubernetes.io/projected/362c9035-e63c-425b-9826-a2453cf0aa18-kube-api-access-65v2t\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:50.649159 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:50.649127    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2299b666-7ae6-452c-bde8-203a6de2e0e2" path="/var/lib/kubelet/pods/2299b666-7ae6-452c-bde8-203a6de2e0e2/volumes"
Oct 30 19:41:50.990764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:50.988678    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ca9d14-qpd57" event={"ID":"362c9035-e63c-425b-9826-a2453cf0aa18","Type":"ContainerDied","Data":"28edb4f76cc7f4e91adecddeebb8063e4187896a89cd0bc0a8d1cd1bddeae067"}
Oct 30 19:41:50.990764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:50.988799    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28edb4f76cc7f4e91adecddeebb8063e4187896a89cd0bc0a8d1cd1bddeae067"
Oct 30 19:41:50.990764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:50.988872    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ca9d14-qpd57"
Oct 30 19:41:51.627144 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:51.627059    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-47110f/test-model-47110f-kserve-bd876684f-4x8lh"]
Oct 30 19:41:51.629466 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:51.629422    5043 cpu_manager.go:414] "containerMap: removing container" podUID="2299b666-7ae6-452c-bde8-203a6de2e0e2" containerName="istio-proxy"
Oct 30 19:41:51.629666 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:51.629648    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2299b666-7ae6-452c-bde8-203a6de2e0e2" containerName="istio-proxy"
Oct 30 19:41:51.629844 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:51.629819    5043 cpu_manager.go:414] "containerMap: removing container" podUID="2299b666-7ae6-452c-bde8-203a6de2e0e2" containerName="istio-proxy"
Oct 30 19:41:51.629965 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:51.629951    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2299b666-7ae6-452c-bde8-203a6de2e0e2" containerName="istio-proxy"
Oct 30 19:41:51.630105 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:51.630092    5043 cpu_manager.go:414] "containerMap: removing container" podUID="362c9035-e63c-425b-9826-a2453cf0aa18" containerName="revoke-keys"
Oct 30 19:41:51.630259 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:51.630245    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="362c9035-e63c-425b-9826-a2453cf0aa18" containerName="revoke-keys"
Oct 30 19:41:51.630603 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:51.630585    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="362c9035-e63c-425b-9826-a2453cf0aa18" containerName="revoke-keys"
Oct 30 19:41:51.630779 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:51.630727    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="2299b666-7ae6-452c-bde8-203a6de2e0e2" containerName="istio-proxy"
Oct 30 19:41:51.630897 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:51.630885    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="2299b666-7ae6-452c-bde8-203a6de2e0e2" containerName="istio-proxy"
Oct 30 19:41:51.643071 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:51.642168    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-47110f/test-model-47110f-kserve-bd876684f-4x8lh"
Oct 30 19:41:51.647978 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:51.647954    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-47110f\"/\"default-dockercfg-sksds\""
Oct 30 19:41:51.653598 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:51.652829    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-47110f\"/\"openshift-service-ca.crt\""
Oct 30 19:41:51.653598 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:51.652943    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-47110f\"/\"kube-root-ca.crt\""
Oct 30 19:41:51.653598 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:51.653533    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-47110f\"/\"test-model-47110f-kserve-self-signed-certs\""
Oct 30 19:41:51.666793 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:51.666717    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-47110f/test-model-47110f-kserve-bd876684f-4x8lh"]
Oct 30 19:41:51.718337 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:51.717242    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2528146c-d4d9-43aa-80a3-c77430db2bf6-model-cache\") pod \"test-model-47110f-kserve-bd876684f-4x8lh\" (UID: \"2528146c-d4d9-43aa-80a3-c77430db2bf6\") " pod="ai-tenant-e2e-inf-b-47110f/test-model-47110f-kserve-bd876684f-4x8lh"
Oct 30 19:41:51.718337 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:51.717285    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2zqq\" (UniqueName: \"kubernetes.io/projected/2528146c-d4d9-43aa-80a3-c77430db2bf6-kube-api-access-r2zqq\") pod \"test-model-47110f-kserve-bd876684f-4x8lh\" (UID: \"2528146c-d4d9-43aa-80a3-c77430db2bf6\") " pod="ai-tenant-e2e-inf-b-47110f/test-model-47110f-kserve-bd876684f-4x8lh"
Oct 30 19:41:51.718337 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:51.717318    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2528146c-d4d9-43aa-80a3-c77430db2bf6-tls-certs\") pod \"test-model-47110f-kserve-bd876684f-4x8lh\" (UID: \"2528146c-d4d9-43aa-80a3-c77430db2bf6\") " pod="ai-tenant-e2e-inf-b-47110f/test-model-47110f-kserve-bd876684f-4x8lh"
Oct 30 19:41:51.718337 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:51.717416    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2528146c-d4d9-43aa-80a3-c77430db2bf6-home\") pod \"test-model-47110f-kserve-bd876684f-4x8lh\" (UID: \"2528146c-d4d9-43aa-80a3-c77430db2bf6\") " pod="ai-tenant-e2e-inf-b-47110f/test-model-47110f-kserve-bd876684f-4x8lh"
Oct 30 19:41:51.718337 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:51.717595    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2528146c-d4d9-43aa-80a3-c77430db2bf6-dshm\") pod \"test-model-47110f-kserve-bd876684f-4x8lh\" (UID: \"2528146c-d4d9-43aa-80a3-c77430db2bf6\") " pod="ai-tenant-e2e-inf-b-47110f/test-model-47110f-kserve-bd876684f-4x8lh"
Oct 30 19:41:51.718337 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:51.717622    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2528146c-d4d9-43aa-80a3-c77430db2bf6-tmp-dir\") pod \"test-model-47110f-kserve-bd876684f-4x8lh\" (UID: \"2528146c-d4d9-43aa-80a3-c77430db2bf6\") " pod="ai-tenant-e2e-inf-b-47110f/test-model-47110f-kserve-bd876684f-4x8lh"
Oct 30 19:41:51.868175 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:51.863261    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2528146c-d4d9-43aa-80a3-c77430db2bf6-dshm\") pod \"test-model-47110f-kserve-bd876684f-4x8lh\" (UID: \"2528146c-d4d9-43aa-80a3-c77430db2bf6\") " pod="ai-tenant-e2e-inf-b-47110f/test-model-47110f-kserve-bd876684f-4x8lh"
Oct 30 19:41:51.868175 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:51.863326    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2528146c-d4d9-43aa-80a3-c77430db2bf6-tmp-dir\") pod \"test-model-47110f-kserve-bd876684f-4x8lh\" (UID: \"2528146c-d4d9-43aa-80a3-c77430db2bf6\") " pod="ai-tenant-e2e-inf-b-47110f/test-model-47110f-kserve-bd876684f-4x8lh"
Oct 30 19:41:51.868175 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:51.863498    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2528146c-d4d9-43aa-80a3-c77430db2bf6-model-cache\") pod \"test-model-47110f-kserve-bd876684f-4x8lh\" (UID: \"2528146c-d4d9-43aa-80a3-c77430db2bf6\") " pod="ai-tenant-e2e-inf-b-47110f/test-model-47110f-kserve-bd876684f-4x8lh"
Oct 30 19:41:51.868175 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:51.863524    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2zqq\" (UniqueName: \"kubernetes.io/projected/2528146c-d4d9-43aa-80a3-c77430db2bf6-kube-api-access-r2zqq\") pod \"test-model-47110f-kserve-bd876684f-4x8lh\" (UID: \"2528146c-d4d9-43aa-80a3-c77430db2bf6\") " pod="ai-tenant-e2e-inf-b-47110f/test-model-47110f-kserve-bd876684f-4x8lh"
Oct 30 19:41:51.868175 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:51.863557    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2528146c-d4d9-43aa-80a3-c77430db2bf6-tls-certs\") pod \"test-model-47110f-kserve-bd876684f-4x8lh\" (UID: \"2528146c-d4d9-43aa-80a3-c77430db2bf6\") " pod="ai-tenant-e2e-inf-b-47110f/test-model-47110f-kserve-bd876684f-4x8lh"
Oct 30 19:41:51.868175 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:51.863663    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2528146c-d4d9-43aa-80a3-c77430db2bf6-home\") pod \"test-model-47110f-kserve-bd876684f-4x8lh\" (UID: \"2528146c-d4d9-43aa-80a3-c77430db2bf6\") " pod="ai-tenant-e2e-inf-b-47110f/test-model-47110f-kserve-bd876684f-4x8lh"
Oct 30 19:41:51.868175 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:51.864350    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2528146c-d4d9-43aa-80a3-c77430db2bf6-home\") pod \"test-model-47110f-kserve-bd876684f-4x8lh\" (UID: \"2528146c-d4d9-43aa-80a3-c77430db2bf6\") " pod="ai-tenant-e2e-inf-b-47110f/test-model-47110f-kserve-bd876684f-4x8lh"
Oct 30 19:41:51.868175 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:51.864987    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2528146c-d4d9-43aa-80a3-c77430db2bf6-model-cache\") pod \"test-model-47110f-kserve-bd876684f-4x8lh\" (UID: \"2528146c-d4d9-43aa-80a3-c77430db2bf6\") " pod="ai-tenant-e2e-inf-b-47110f/test-model-47110f-kserve-bd876684f-4x8lh"
Oct 30 19:41:51.868175 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:51.865233    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2528146c-d4d9-43aa-80a3-c77430db2bf6-tmp-dir\") pod \"test-model-47110f-kserve-bd876684f-4x8lh\" (UID: \"2528146c-d4d9-43aa-80a3-c77430db2bf6\") " pod="ai-tenant-e2e-inf-b-47110f/test-model-47110f-kserve-bd876684f-4x8lh"
Oct 30 19:41:51.979829 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:51.975453    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2528146c-d4d9-43aa-80a3-c77430db2bf6-tls-certs\") pod \"test-model-47110f-kserve-bd876684f-4x8lh\" (UID: \"2528146c-d4d9-43aa-80a3-c77430db2bf6\") " pod="ai-tenant-e2e-inf-b-47110f/test-model-47110f-kserve-bd876684f-4x8lh"
Oct 30 19:41:51.979829 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:51.976098    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2528146c-d4d9-43aa-80a3-c77430db2bf6-dshm\") pod \"test-model-47110f-kserve-bd876684f-4x8lh\" (UID: \"2528146c-d4d9-43aa-80a3-c77430db2bf6\") " pod="ai-tenant-e2e-inf-b-47110f/test-model-47110f-kserve-bd876684f-4x8lh"
Oct 30 19:41:51.979829 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:51.979659    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc"
Oct 30 19:41:51.992357 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:51.985590    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2zqq\" (UniqueName: \"kubernetes.io/projected/2528146c-d4d9-43aa-80a3-c77430db2bf6-kube-api-access-r2zqq\") pod \"test-model-47110f-kserve-bd876684f-4x8lh\" (UID: \"2528146c-d4d9-43aa-80a3-c77430db2bf6\") " pod="ai-tenant-e2e-inf-b-47110f/test-model-47110f-kserve-bd876684f-4x8lh"
Oct 30 19:41:52.003307 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:52.003263    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-47110f/test-model-47110f-kserve-bd876684f-4x8lh"
Oct 30 19:41:52.005643 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:52.004762    5043 generic.go:356] "Generic (PLEG): container finished" podID="b34f07ce-6edf-4493-b349-ad7916bb93aa" containerID="d1b0d067fd0d220353deb56c12459de13b426a871771643c434535c21c1ae5ff" exitCode=0
Oct 30 19:41:52.005643 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:52.005023    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc" event={"ID":"b34f07ce-6edf-4493-b349-ad7916bb93aa","Type":"ContainerDied","Data":"d1b0d067fd0d220353deb56c12459de13b426a871771643c434535c21c1ae5ff"}
Oct 30 19:41:52.005643 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:52.005083    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc" event={"ID":"b34f07ce-6edf-4493-b349-ad7916bb93aa","Type":"ContainerDied","Data":"969cf7e7b03153a1e88d9f0cd4ce722fbee8d70be4c70e26d82b1b7710e9778c"}
Oct 30 19:41:52.005643 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:52.005117    5043 scope.go:122] "RemoveContainer" containerID="d1b0d067fd0d220353deb56c12459de13b426a871771643c434535c21c1ae5ff"
Oct 30 19:41:52.005643 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:52.005284    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc"
Oct 30 19:41:52.036204 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:52.036178    5043 scope.go:122] "RemoveContainer" containerID="35d3bb6a619b8fd1b7a20cd5648be3a3d97cbbd1ee8170fd1ebc766131c7f752"
Oct 30 19:41:52.054111 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:52.054086    5043 scope.go:122] "RemoveContainer" containerID="d1b0d067fd0d220353deb56c12459de13b426a871771643c434535c21c1ae5ff"
Oct 30 19:41:52.054506 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:41:52.054481    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1b0d067fd0d220353deb56c12459de13b426a871771643c434535c21c1ae5ff\": container with ID starting with d1b0d067fd0d220353deb56c12459de13b426a871771643c434535c21c1ae5ff not found: ID does not exist" containerID="d1b0d067fd0d220353deb56c12459de13b426a871771643c434535c21c1ae5ff"
Oct 30 19:41:52.054606 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:52.054515    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1b0d067fd0d220353deb56c12459de13b426a871771643c434535c21c1ae5ff"} err="failed to get container status \"d1b0d067fd0d220353deb56c12459de13b426a871771643c434535c21c1ae5ff\": rpc error: code = NotFound desc = could not find container \"d1b0d067fd0d220353deb56c12459de13b426a871771643c434535c21c1ae5ff\": container with ID starting with d1b0d067fd0d220353deb56c12459de13b426a871771643c434535c21c1ae5ff not found: ID does not exist"
Oct 30 19:41:52.054606 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:52.054534    5043 scope.go:122] "RemoveContainer" containerID="35d3bb6a619b8fd1b7a20cd5648be3a3d97cbbd1ee8170fd1ebc766131c7f752"
Oct 30 19:41:52.054819 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:41:52.054797    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35d3bb6a619b8fd1b7a20cd5648be3a3d97cbbd1ee8170fd1ebc766131c7f752\": container with ID starting with 35d3bb6a619b8fd1b7a20cd5648be3a3d97cbbd1ee8170fd1ebc766131c7f752 not found: ID does not exist" containerID="35d3bb6a619b8fd1b7a20cd5648be3a3d97cbbd1ee8170fd1ebc766131c7f752"
Oct 30 19:41:52.054921 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:52.054827    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35d3bb6a619b8fd1b7a20cd5648be3a3d97cbbd1ee8170fd1ebc766131c7f752"} err="failed to get container status \"35d3bb6a619b8fd1b7a20cd5648be3a3d97cbbd1ee8170fd1ebc766131c7f752\": rpc error: code = NotFound desc = could not find container \"35d3bb6a619b8fd1b7a20cd5648be3a3d97cbbd1ee8170fd1ebc766131c7f752\": container with ID starting with 35d3bb6a619b8fd1b7a20cd5648be3a3d97cbbd1ee8170fd1ebc766131c7f752 not found: ID does not exist"
Oct 30 19:41:52.080163 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:52.080108    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b34f07ce-6edf-4493-b349-ad7916bb93aa-istio-token\" (UniqueName: \"kubernetes.io/projected/b34f07ce-6edf-4493-b349-ad7916bb93aa-istio-token\") pod \"b34f07ce-6edf-4493-b349-ad7916bb93aa\" (UID: \"b34f07ce-6edf-4493-b349-ad7916bb93aa\") "
Oct 30 19:41:52.080163 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:52.080171    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b34f07ce-6edf-4493-b349-ad7916bb93aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b34f07ce-6edf-4493-b349-ad7916bb93aa-workload-certs\") pod \"b34f07ce-6edf-4493-b349-ad7916bb93aa\" (UID: \"b34f07ce-6edf-4493-b349-ad7916bb93aa\") "
Oct 30 19:41:52.080464 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:52.080217    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b34f07ce-6edf-4493-b349-ad7916bb93aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b34f07ce-6edf-4493-b349-ad7916bb93aa-credential-socket\") pod \"b34f07ce-6edf-4493-b349-ad7916bb93aa\" (UID: \"b34f07ce-6edf-4493-b349-ad7916bb93aa\") "
Oct 30 19:41:52.080464 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:52.080239    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b34f07ce-6edf-4493-b349-ad7916bb93aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b34f07ce-6edf-4493-b349-ad7916bb93aa-workload-socket\") pod \"b34f07ce-6edf-4493-b349-ad7916bb93aa\" (UID: \"b34f07ce-6edf-4493-b349-ad7916bb93aa\") "
Oct 30 19:41:52.081122 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:52.081093    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b34f07ce-6edf-4493-b349-ad7916bb93aa-credential-socket" pod "b34f07ce-6edf-4493-b349-ad7916bb93aa" (UID: "b34f07ce-6edf-4493-b349-ad7916bb93aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:52.081122 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:52.081103    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b34f07ce-6edf-4493-b349-ad7916bb93aa-workload-certs" pod "b34f07ce-6edf-4493-b349-ad7916bb93aa" (UID: "b34f07ce-6edf-4493-b349-ad7916bb93aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:52.081295 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:52.081162    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b34f07ce-6edf-4493-b349-ad7916bb93aa-workload-socket" pod "b34f07ce-6edf-4493-b349-ad7916bb93aa" (UID: "b34f07ce-6edf-4493-b349-ad7916bb93aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:52.084402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:52.084371    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b34f07ce-6edf-4493-b349-ad7916bb93aa-istio-token" pod "b34f07ce-6edf-4493-b349-ad7916bb93aa" (UID: "b34f07ce-6edf-4493-b349-ad7916bb93aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:41:52.181521 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:52.181487    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b34f07ce-6edf-4493-b349-ad7916bb93aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b34f07ce-6edf-4493-b349-ad7916bb93aa-istio-envoy\") pod \"b34f07ce-6edf-4493-b349-ad7916bb93aa\" (UID: \"b34f07ce-6edf-4493-b349-ad7916bb93aa\") "
Oct 30 19:41:52.181697 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:52.181682    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b34f07ce-6edf-4493-b349-ad7916bb93aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b34f07ce-6edf-4493-b349-ad7916bb93aa-istio-data\") pod \"b34f07ce-6edf-4493-b349-ad7916bb93aa\" (UID: \"b34f07ce-6edf-4493-b349-ad7916bb93aa\") "
Oct 30 19:41:52.181818 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:52.181798    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b34f07ce-6edf-4493-b349-ad7916bb93aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b34f07ce-6edf-4493-b349-ad7916bb93aa-istiod-ca-cert\") pod \"b34f07ce-6edf-4493-b349-ad7916bb93aa\" (UID: \"b34f07ce-6edf-4493-b349-ad7916bb93aa\") "
Oct 30 19:41:52.181949 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:52.181932    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b34f07ce-6edf-4493-b349-ad7916bb93aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b34f07ce-6edf-4493-b349-ad7916bb93aa-istio-podinfo\") pod \"b34f07ce-6edf-4493-b349-ad7916bb93aa\" (UID: \"b34f07ce-6edf-4493-b349-ad7916bb93aa\") "
Oct 30 19:41:52.182012 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:52.181976    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b34f07ce-6edf-4493-b349-ad7916bb93aa-kube-api-access-2q2gw\" (UniqueName: \"kubernetes.io/projected/b34f07ce-6edf-4493-b349-ad7916bb93aa-kube-api-access-2q2gw\") pod \"b34f07ce-6edf-4493-b349-ad7916bb93aa\" (UID: \"b34f07ce-6edf-4493-b349-ad7916bb93aa\") "
Oct 30 19:41:52.182012 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:52.181974    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b34f07ce-6edf-4493-b349-ad7916bb93aa-istio-data" pod "b34f07ce-6edf-4493-b349-ad7916bb93aa" (UID: "b34f07ce-6edf-4493-b349-ad7916bb93aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:52.182150 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:52.182128    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b34f07ce-6edf-4493-b349-ad7916bb93aa-istiod-ca-cert" pod "b34f07ce-6edf-4493-b349-ad7916bb93aa" (UID: "b34f07ce-6edf-4493-b349-ad7916bb93aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:41:52.182855 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:52.182828    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b34f07ce-6edf-4493-b349-ad7916bb93aa-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:52.182855 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:52.182857    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b34f07ce-6edf-4493-b349-ad7916bb93aa-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:52.183085 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:52.182876    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b34f07ce-6edf-4493-b349-ad7916bb93aa-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:52.183085 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:52.182894    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b34f07ce-6edf-4493-b349-ad7916bb93aa-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:52.183085 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:52.182906    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b34f07ce-6edf-4493-b349-ad7916bb93aa-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:52.183085 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:52.182918    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b34f07ce-6edf-4493-b349-ad7916bb93aa-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:52.184014 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:52.183991    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b34f07ce-6edf-4493-b349-ad7916bb93aa-istio-envoy" pod "b34f07ce-6edf-4493-b349-ad7916bb93aa" (UID: "b34f07ce-6edf-4493-b349-ad7916bb93aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:52.184431 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:52.184406    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b34f07ce-6edf-4493-b349-ad7916bb93aa-kube-api-access-2q2gw" pod "b34f07ce-6edf-4493-b349-ad7916bb93aa" (UID: "b34f07ce-6edf-4493-b349-ad7916bb93aa"). InnerVolumeSpecName "kube-api-access-2q2gw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:41:52.184598 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:52.184574    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b34f07ce-6edf-4493-b349-ad7916bb93aa-istio-podinfo" pod "b34f07ce-6edf-4493-b349-ad7916bb93aa" (UID: "b34f07ce-6edf-4493-b349-ad7916bb93aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:41:52.284884 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:52.284780    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b34f07ce-6edf-4493-b349-ad7916bb93aa-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:52.284884 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:52.284819    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q2gw\" (UniqueName: \"kubernetes.io/projected/b34f07ce-6edf-4493-b349-ad7916bb93aa-kube-api-access-2q2gw\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:52.284884 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:52.284834    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b34f07ce-6edf-4493-b349-ad7916bb93aa-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:41:52.323291 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:52.323256    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-47110f/test-model-47110f-kserve-bd876684f-4x8lh"]
Oct 30 19:41:52.323672 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:41:52.323651    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2528146c_d4d9_43aa_80a3_c77430db2bf6.slice/crio-a43b9b0a4ed4c26cbf3fa11040dd329bd4ac7369f0bf404cea34691ec8f05d12 WatchSource:0}: Error finding container a43b9b0a4ed4c26cbf3fa11040dd329bd4ac7369f0bf404cea34691ec8f05d12: Status 404 returned error can't find the container with id a43b9b0a4ed4c26cbf3fa11040dd329bd4ac7369f0bf404cea34691ec8f05d12
Oct 30 19:41:52.431969 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:52.431862    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc"]
Oct 30 19:41:52.437859 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:52.437825    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-59f017-openshift-default-ddc89b68c-lshmc"]
Oct 30 19:41:52.648174 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:52.648085    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b34f07ce-6edf-4493-b349-ad7916bb93aa" path="/var/lib/kubelet/pods/b34f07ce-6edf-4493-b349-ad7916bb93aa/volumes"
Oct 30 19:41:53.016712 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:53.016677    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-47110f/test-model-47110f-kserve-bd876684f-4x8lh" event={"ID":"2528146c-d4d9-43aa-80a3-c77430db2bf6","Type":"ContainerStarted","Data":"38c0469a824312283aec48f464e153ca524db6b6bae3840fe6e3f6a27a8eb1e9"}
Oct 30 19:41:53.016712 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:53.016719    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-47110f/test-model-47110f-kserve-bd876684f-4x8lh" event={"ID":"2528146c-d4d9-43aa-80a3-c77430db2bf6","Type":"ContainerStarted","Data":"a43b9b0a4ed4c26cbf3fa11040dd329bd4ac7369f0bf404cea34691ec8f05d12"}
Oct 30 19:41:53.074551 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:53.074485    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-47110f/test-model-47110f-kserve-bd876684f-4x8lh" podStartSLOduration=2.07446617 podStartE2EDuration="2.07446617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:53.069990155 +0000 UTC m=+1532.581050125" watchObservedRunningTime="2026-10-30 19:41:53.07446617 +0000 UTC m=+1532.585526135"
Oct 30 19:41:54.003535 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:54.003497    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-47110f/test-model-47110f-kserve-bd876684f-4x8lh"
Oct 30 19:41:54.980778 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:41:54.980005    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:41:56.745362 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:56.745321    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ca9d14-746f5b5598-6jt2z"]
Oct 30 19:41:56.764462 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:56.764425    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca9d14-5c478d7cc5-wtmp4"]
Oct 30 19:41:57.078816 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:57.071024    5043 generic.go:356] "Generic (PLEG): container finished" podID="bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e" containerID="6c16073da015e4edf494cce7836e9ace33fde259c9c9db2a491e02bdbfce47f1" exitCode=0
Oct 30 19:41:57.078816 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:57.071341    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46" event={"ID":"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e","Type":"ContainerDied","Data":"6c16073da015e4edf494cce7836e9ace33fde259c9c9db2a491e02bdbfce47f1"}
Oct 30 19:41:57.078816 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:57.072178    5043 scope.go:122] "RemoveContainer" containerID="6c16073da015e4edf494cce7836e9ace33fde259c9c9db2a491e02bdbfce47f1"
Oct 30 19:41:57.078816 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:57.077374    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:41:57.078816 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:57.077611    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:41:57.078816 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:57.077668    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:41:58.133360 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:58.133309    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46" event={"ID":"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e","Type":"ContainerStarted","Data":"313d3de3030cb40c4cc6193cd87d37b73de4c2b7c754f231ecf326eeb0150c1f"}
Oct 30 19:41:58.253748 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:58.253695    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46"
Oct 30 19:41:58.254935 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:58.254906    5043 patch_prober.go:29] interesting pod/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body=
Oct 30 19:41:58.255064 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:58.254967    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46" podUID="bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused"
Oct 30 19:41:59.253790 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:59.253729    5043 patch_prober.go:29] interesting pod/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body=
Oct 30 19:41:59.254386 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:41:59.253811    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46" podUID="bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused"
Oct 30 19:42:00.194944 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:00.194907    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46"
Oct 30 19:42:00.261925 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:00.261876    5043 patch_prober.go:29] interesting pod/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body=
Oct 30 19:42:00.262512 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:00.262465    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46" podUID="bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused"
Oct 30 19:42:00.285420 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:00.284656    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ca9d14-qpd57"]
Oct 30 19:42:00.404763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:00.404187    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ca9d14-qpd57"]
Oct 30 19:42:00.649968 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:00.649936    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="362c9035-e63c-425b-9826-a2453cf0aa18" path="/var/lib/kubelet/pods/362c9035-e63c-425b-9826-a2453cf0aa18/volumes"
Oct 30 19:42:01.254274 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:01.254224    5043 patch_prober.go:29] interesting pod/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body=
Oct 30 19:42:01.254555 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:01.254525    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46" podUID="bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused"
Oct 30 19:42:01.746269 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:01.746203    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca9d14-746f5b5598-6jt2z" podUID="28d04237-1b82-4642-b811-5bd3cd6e2068" containerName="payload-processing" containerID="cri-o://b4bac5d3ae6fad4260ba95633b7a1157f04aa5a2294b863aaaf6216e03b889b5" gracePeriod=25
Oct 30 19:42:01.767428 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:01.766805    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca9d14-5c478d7cc5-wtmp4" podUID="aaf1c38e-171d-4874-8a38-9f0e03deacce" containerName="payload-pre-processing" containerID="cri-o://2b97ea60a086796e4953cabc498fc45cca0017dab4cc37d4370e0b288ab9152f" gracePeriod=25
Oct 30 19:42:02.008820 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:02.003938    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-47110f/test-model-47110f-kserve-bd876684f-4x8lh"
Oct 30 19:42:02.118050 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:02.117696    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-47110f/test-model-47110f-kserve-bd876684f-4x8lh"
Oct 30 19:42:02.254873 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:02.254499    5043 patch_prober.go:29] interesting pod/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body=
Oct 30 19:42:02.254873 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:02.254598    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46" podUID="bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused"
Oct 30 19:42:02.339239 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:02.339152    5043 generic.go:356] "Generic (PLEG): container finished" podID="28d04237-1b82-4642-b811-5bd3cd6e2068" containerID="b4bac5d3ae6fad4260ba95633b7a1157f04aa5a2294b863aaaf6216e03b889b5" exitCode=0
Oct 30 19:42:02.339750 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:02.339312    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca9d14-746f5b5598-6jt2z" event={"ID":"28d04237-1b82-4642-b811-5bd3cd6e2068","Type":"ContainerDied","Data":"b4bac5d3ae6fad4260ba95633b7a1157f04aa5a2294b863aaaf6216e03b889b5"}
Oct 30 19:42:02.346542 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:02.345257    5043 generic.go:356] "Generic (PLEG): container finished" podID="aaf1c38e-171d-4874-8a38-9f0e03deacce" containerID="2b97ea60a086796e4953cabc498fc45cca0017dab4cc37d4370e0b288ab9152f" exitCode=0
Oct 30 19:42:02.351940 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:02.346801    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca9d14-5c478d7cc5-wtmp4" event={"ID":"aaf1c38e-171d-4874-8a38-9f0e03deacce","Type":"ContainerDied","Data":"2b97ea60a086796e4953cabc498fc45cca0017dab4cc37d4370e0b288ab9152f"}
Oct 30 19:42:02.433916 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:02.433892    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-47110f/test-model-47110f-kserve-bd876684f-4x8lh"
Oct 30 19:42:02.439246 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:02.439223    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca9d14-5c478d7cc5-wtmp4"
Oct 30 19:42:02.547122 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:02.546718    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaf1c38e-171d-4874-8a38-9f0e03deacce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaf1c38e-171d-4874-8a38-9f0e03deacce-plugins-config-volume\") pod \"aaf1c38e-171d-4874-8a38-9f0e03deacce\" (UID: \"aaf1c38e-171d-4874-8a38-9f0e03deacce\") "
Oct 30 19:42:02.547122 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:02.547002    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaf1c38e-171d-4874-8a38-9f0e03deacce-kube-api-access-c9pbn\" (UniqueName: \"kubernetes.io/projected/aaf1c38e-171d-4874-8a38-9f0e03deacce-kube-api-access-c9pbn\") pod \"aaf1c38e-171d-4874-8a38-9f0e03deacce\" (UID: \"aaf1c38e-171d-4874-8a38-9f0e03deacce\") "
Oct 30 19:42:02.549169 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:02.549138    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaf1c38e-171d-4874-8a38-9f0e03deacce-plugins-config-volume" pod "aaf1c38e-171d-4874-8a38-9f0e03deacce" (UID: "aaf1c38e-171d-4874-8a38-9f0e03deacce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:42:02.553009 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:02.552923    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaf1c38e-171d-4874-8a38-9f0e03deacce-kube-api-access-c9pbn" pod "aaf1c38e-171d-4874-8a38-9f0e03deacce" (UID: "aaf1c38e-171d-4874-8a38-9f0e03deacce"). InnerVolumeSpecName "kube-api-access-c9pbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:02.652768 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:02.650209    5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaf1c38e-171d-4874-8a38-9f0e03deacce-plugins-config-volume\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:02.652768 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:02.650243    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9pbn\" (UniqueName: \"kubernetes.io/projected/aaf1c38e-171d-4874-8a38-9f0e03deacce-kube-api-access-c9pbn\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:02.670793 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:02.670573    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca9d14-746f5b5598-6jt2z"
Oct 30 19:42:02.751819 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:02.751778    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28d04237-1b82-4642-b811-5bd3cd6e2068-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28d04237-1b82-4642-b811-5bd3cd6e2068-plugins-config-volume\") pod \"28d04237-1b82-4642-b811-5bd3cd6e2068\" (UID: \"28d04237-1b82-4642-b811-5bd3cd6e2068\") "
Oct 30 19:42:02.752766 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:02.752176    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28d04237-1b82-4642-b811-5bd3cd6e2068-plugins-config-volume" pod "28d04237-1b82-4642-b811-5bd3cd6e2068" (UID: "28d04237-1b82-4642-b811-5bd3cd6e2068"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:42:02.752766 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:02.752451    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28d04237-1b82-4642-b811-5bd3cd6e2068-kube-api-access-8x7rn\" (UniqueName: \"kubernetes.io/projected/28d04237-1b82-4642-b811-5bd3cd6e2068-kube-api-access-8x7rn\") pod \"28d04237-1b82-4642-b811-5bd3cd6e2068\" (UID: \"28d04237-1b82-4642-b811-5bd3cd6e2068\") "
Oct 30 19:42:02.753452 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:02.753426    5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28d04237-1b82-4642-b811-5bd3cd6e2068-plugins-config-volume\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:02.807810 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:02.805979    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28d04237-1b82-4642-b811-5bd3cd6e2068-kube-api-access-8x7rn" pod "28d04237-1b82-4642-b811-5bd3cd6e2068" (UID: "28d04237-1b82-4642-b811-5bd3cd6e2068"). InnerVolumeSpecName "kube-api-access-8x7rn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:02.873765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:02.873324    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x7rn\" (UniqueName: \"kubernetes.io/projected/28d04237-1b82-4642-b811-5bd3cd6e2068-kube-api-access-8x7rn\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:03.254638 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:03.254597    5043 patch_prober.go:29] interesting pod/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body=
Oct 30 19:42:03.254851 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:03.254663    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46" podUID="bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused"
Oct 30 19:42:03.388414 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:03.388378    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca9d14-5c478d7cc5-wtmp4"
Oct 30 19:42:03.388623 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:03.388453    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca9d14-5c478d7cc5-wtmp4" event={"ID":"aaf1c38e-171d-4874-8a38-9f0e03deacce","Type":"ContainerDied","Data":"7dc704bd1e62d6a8fb2532b0be20b50f46d193aa3ab55464450d862cd0a25ad0"}
Oct 30 19:42:03.388623 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:03.388567    5043 scope.go:122] "RemoveContainer" containerID="2b97ea60a086796e4953cabc498fc45cca0017dab4cc37d4370e0b288ab9152f"
Oct 30 19:42:03.392845 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:03.392816    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca9d14-746f5b5598-6jt2z" event={"ID":"28d04237-1b82-4642-b811-5bd3cd6e2068","Type":"ContainerDied","Data":"d867b5b6a513f9ce3a48c3081c73239ac96886f215b6a7eaf9a45ec77f32508b"}
Oct 30 19:42:03.392976 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:03.392865    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca9d14-746f5b5598-6jt2z"
Oct 30 19:42:03.403714 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:03.403694    5043 scope.go:122] "RemoveContainer" containerID="b4bac5d3ae6fad4260ba95633b7a1157f04aa5a2294b863aaaf6216e03b889b5"
Oct 30 19:42:03.416189 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:03.416164    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca9d14-5c478d7cc5-wtmp4"]
Oct 30 19:42:03.425611 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:03.424270    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca9d14-5c478d7cc5-wtmp4"]
Oct 30 19:42:03.429931 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:03.429906    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ca9d14-746f5b5598-6jt2z"]
Oct 30 19:42:03.438985 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:03.438959    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ca9d14-746f5b5598-6jt2z"]
Oct 30 19:42:04.262176 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:04.262147    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46"
Oct 30 19:42:04.263460 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:04.263437    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46"
Oct 30 19:42:04.649987 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:04.649907    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28d04237-1b82-4642-b811-5bd3cd6e2068" path="/var/lib/kubelet/pods/28d04237-1b82-4642-b811-5bd3cd6e2068/volumes"
Oct 30 19:42:04.650449 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:04.650434    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaf1c38e-171d-4874-8a38-9f0e03deacce" path="/var/lib/kubelet/pods/aaf1c38e-171d-4874-8a38-9f0e03deacce/volumes"
Oct 30 19:42:04.856964 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:04.856920    5043 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:42:04.857179 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:04.856987    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt" podUID="cffde88e-6bb5-4c7c-b408-bbf25e17f381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:07.808635 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:07.808595    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt"]
Oct 30 19:42:07.825177 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:07.808818    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt" podUID="cffde88e-6bb5-4c7c-b408-bbf25e17f381" containerName="istio-proxy" containerID="cri-o://28d793449ee82dc6b14147dfd838d174b5e0f7c7ab9978608299f0c911c6115e" gracePeriod=30
Oct 30 19:42:08.435567 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.435542    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt"
Oct 30 19:42:08.452036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.452001    5043 generic.go:356] "Generic (PLEG): container finished" podID="cffde88e-6bb5-4c7c-b408-bbf25e17f381" containerID="28d793449ee82dc6b14147dfd838d174b5e0f7c7ab9978608299f0c911c6115e" exitCode=0
Oct 30 19:42:08.452221 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.452052    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt"
Oct 30 19:42:08.452221 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.452085    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt" event={"ID":"cffde88e-6bb5-4c7c-b408-bbf25e17f381","Type":"ContainerDied","Data":"28d793449ee82dc6b14147dfd838d174b5e0f7c7ab9978608299f0c911c6115e"}
Oct 30 19:42:08.452221 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.452146    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt" event={"ID":"cffde88e-6bb5-4c7c-b408-bbf25e17f381","Type":"ContainerDied","Data":"1e47a59df30cf885048696da9a6620b682978d514c43a4cd74992656f3a8a1ba"}
Oct 30 19:42:08.452221 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.452174    5043 scope.go:122] "RemoveContainer" containerID="28d793449ee82dc6b14147dfd838d174b5e0f7c7ab9978608299f0c911c6115e"
Oct 30 19:42:08.463853 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.463830    5043 scope.go:122] "RemoveContainer" containerID="28d793449ee82dc6b14147dfd838d174b5e0f7c7ab9978608299f0c911c6115e"
Oct 30 19:42:08.464211 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:42:08.464191    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28d793449ee82dc6b14147dfd838d174b5e0f7c7ab9978608299f0c911c6115e\": container with ID starting with 28d793449ee82dc6b14147dfd838d174b5e0f7c7ab9978608299f0c911c6115e not found: ID does not exist" containerID="28d793449ee82dc6b14147dfd838d174b5e0f7c7ab9978608299f0c911c6115e"
Oct 30 19:42:08.464285 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.464219    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28d793449ee82dc6b14147dfd838d174b5e0f7c7ab9978608299f0c911c6115e"} err="failed to get container status \"28d793449ee82dc6b14147dfd838d174b5e0f7c7ab9978608299f0c911c6115e\": rpc error: code = NotFound desc = could not find container \"28d793449ee82dc6b14147dfd838d174b5e0f7c7ab9978608299f0c911c6115e\": container with ID starting with 28d793449ee82dc6b14147dfd838d174b5e0f7c7ab9978608299f0c911c6115e not found: ID does not exist"
Oct 30 19:42:08.489534 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.489490    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cffde88e-6bb5-4c7c-b408-bbf25e17f381-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cffde88e-6bb5-4c7c-b408-bbf25e17f381-workload-certs\") pod \"cffde88e-6bb5-4c7c-b408-bbf25e17f381\" (UID: \"cffde88e-6bb5-4c7c-b408-bbf25e17f381\") "
Oct 30 19:42:08.489757 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.489549    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cffde88e-6bb5-4c7c-b408-bbf25e17f381-kube-api-access-7jxpf\" (UniqueName: \"kubernetes.io/projected/cffde88e-6bb5-4c7c-b408-bbf25e17f381-kube-api-access-7jxpf\") pod \"cffde88e-6bb5-4c7c-b408-bbf25e17f381\" (UID: \"cffde88e-6bb5-4c7c-b408-bbf25e17f381\") "
Oct 30 19:42:08.489757 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.489650    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cffde88e-6bb5-4c7c-b408-bbf25e17f381-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cffde88e-6bb5-4c7c-b408-bbf25e17f381-istio-podinfo\") pod \"cffde88e-6bb5-4c7c-b408-bbf25e17f381\" (UID: \"cffde88e-6bb5-4c7c-b408-bbf25e17f381\") "
Oct 30 19:42:08.489757 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.489685    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cffde88e-6bb5-4c7c-b408-bbf25e17f381-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cffde88e-6bb5-4c7c-b408-bbf25e17f381-istio-envoy\") pod \"cffde88e-6bb5-4c7c-b408-bbf25e17f381\" (UID: \"cffde88e-6bb5-4c7c-b408-bbf25e17f381\") "
Oct 30 19:42:08.489946 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.489784    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cffde88e-6bb5-4c7c-b408-bbf25e17f381-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cffde88e-6bb5-4c7c-b408-bbf25e17f381-istio-data\") pod \"cffde88e-6bb5-4c7c-b408-bbf25e17f381\" (UID: \"cffde88e-6bb5-4c7c-b408-bbf25e17f381\") "
Oct 30 19:42:08.490005 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.489979    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cffde88e-6bb5-4c7c-b408-bbf25e17f381-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cffde88e-6bb5-4c7c-b408-bbf25e17f381-credential-socket\") pod \"cffde88e-6bb5-4c7c-b408-bbf25e17f381\" (UID: \"cffde88e-6bb5-4c7c-b408-bbf25e17f381\") "
Oct 30 19:42:08.490097 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.490078    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cffde88e-6bb5-4c7c-b408-bbf25e17f381-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cffde88e-6bb5-4c7c-b408-bbf25e17f381-istiod-ca-cert\") pod \"cffde88e-6bb5-4c7c-b408-bbf25e17f381\" (UID: \"cffde88e-6bb5-4c7c-b408-bbf25e17f381\") "
Oct 30 19:42:08.490167 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.490130    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cffde88e-6bb5-4c7c-b408-bbf25e17f381-istio-token\" (UniqueName: \"kubernetes.io/projected/cffde88e-6bb5-4c7c-b408-bbf25e17f381-istio-token\") pod \"cffde88e-6bb5-4c7c-b408-bbf25e17f381\" (UID: \"cffde88e-6bb5-4c7c-b408-bbf25e17f381\") "
Oct 30 19:42:08.490224 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.490186    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cffde88e-6bb5-4c7c-b408-bbf25e17f381-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cffde88e-6bb5-4c7c-b408-bbf25e17f381-workload-socket\") pod \"cffde88e-6bb5-4c7c-b408-bbf25e17f381\" (UID: \"cffde88e-6bb5-4c7c-b408-bbf25e17f381\") "
Oct 30 19:42:08.490522 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.490489    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cffde88e-6bb5-4c7c-b408-bbf25e17f381-istio-data" pod "cffde88e-6bb5-4c7c-b408-bbf25e17f381" (UID: "cffde88e-6bb5-4c7c-b408-bbf25e17f381"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:42:08.490796 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.490723    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cffde88e-6bb5-4c7c-b408-bbf25e17f381-workload-socket" pod "cffde88e-6bb5-4c7c-b408-bbf25e17f381" (UID: "cffde88e-6bb5-4c7c-b408-bbf25e17f381"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:42:08.490933 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.490910    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cffde88e-6bb5-4c7c-b408-bbf25e17f381-credential-socket" pod "cffde88e-6bb5-4c7c-b408-bbf25e17f381" (UID: "cffde88e-6bb5-4c7c-b408-bbf25e17f381"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:42:08.491015 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.490999    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cffde88e-6bb5-4c7c-b408-bbf25e17f381-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:08.491078 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.491022    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cffde88e-6bb5-4c7c-b408-bbf25e17f381-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:08.491221 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.491199    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cffde88e-6bb5-4c7c-b408-bbf25e17f381-istiod-ca-cert" pod "cffde88e-6bb5-4c7c-b408-bbf25e17f381" (UID: "cffde88e-6bb5-4c7c-b408-bbf25e17f381"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:42:08.491478 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.491426    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cffde88e-6bb5-4c7c-b408-bbf25e17f381-workload-certs" pod "cffde88e-6bb5-4c7c-b408-bbf25e17f381" (UID: "cffde88e-6bb5-4c7c-b408-bbf25e17f381"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:42:08.496067 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.494459    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cffde88e-6bb5-4c7c-b408-bbf25e17f381-istio-envoy" pod "cffde88e-6bb5-4c7c-b408-bbf25e17f381" (UID: "cffde88e-6bb5-4c7c-b408-bbf25e17f381"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:42:08.496067 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.494522    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cffde88e-6bb5-4c7c-b408-bbf25e17f381-kube-api-access-7jxpf" pod "cffde88e-6bb5-4c7c-b408-bbf25e17f381" (UID: "cffde88e-6bb5-4c7c-b408-bbf25e17f381"). InnerVolumeSpecName "kube-api-access-7jxpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:08.508075 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.496455    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cffde88e-6bb5-4c7c-b408-bbf25e17f381-istio-podinfo" pod "cffde88e-6bb5-4c7c-b408-bbf25e17f381" (UID: "cffde88e-6bb5-4c7c-b408-bbf25e17f381"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:42:08.508075 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.497277    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cffde88e-6bb5-4c7c-b408-bbf25e17f381-istio-token" pod "cffde88e-6bb5-4c7c-b408-bbf25e17f381" (UID: "cffde88e-6bb5-4c7c-b408-bbf25e17f381"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:08.593012 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.592978    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cffde88e-6bb5-4c7c-b408-bbf25e17f381-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:08.593012 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.593009    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cffde88e-6bb5-4c7c-b408-bbf25e17f381-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:08.593012 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.593021    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cffde88e-6bb5-4c7c-b408-bbf25e17f381-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:08.593273 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.593029    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cffde88e-6bb5-4c7c-b408-bbf25e17f381-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:08.593273 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.593041    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jxpf\" (UniqueName: \"kubernetes.io/projected/cffde88e-6bb5-4c7c-b408-bbf25e17f381-kube-api-access-7jxpf\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:08.593273 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.593051    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cffde88e-6bb5-4c7c-b408-bbf25e17f381-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:08.593273 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.593061    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cffde88e-6bb5-4c7c-b408-bbf25e17f381-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:08.668725 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.668624    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b623ca-j9hhm"]
Oct 30 19:42:08.669322 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.669306    5043 cpu_manager.go:414] "containerMap: removing container" podUID="b34f07ce-6edf-4493-b349-ad7916bb93aa" containerName="istio-proxy"
Oct 30 19:42:08.669407 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.669322    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b34f07ce-6edf-4493-b349-ad7916bb93aa" containerName="istio-proxy"
Oct 30 19:42:08.669407 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.669355    5043 cpu_manager.go:414] "containerMap: removing container" podUID="b34f07ce-6edf-4493-b349-ad7916bb93aa" containerName="istio-proxy"
Oct 30 19:42:08.669407 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.669363    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b34f07ce-6edf-4493-b349-ad7916bb93aa" containerName="istio-proxy"
Oct 30 19:42:08.669407 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.669386    5043 cpu_manager.go:414] "containerMap: removing container" podUID="cffde88e-6bb5-4c7c-b408-bbf25e17f381" containerName="istio-proxy"
Oct 30 19:42:08.669407 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.669392    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cffde88e-6bb5-4c7c-b408-bbf25e17f381" containerName="istio-proxy"
Oct 30 19:42:08.669407 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.669402    5043 cpu_manager.go:414] "containerMap: removing container" podUID="28d04237-1b82-4642-b811-5bd3cd6e2068" containerName="payload-processing"
Oct 30 19:42:08.669407 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.669406    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28d04237-1b82-4642-b811-5bd3cd6e2068" containerName="payload-processing"
Oct 30 19:42:08.669854 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.669420    5043 cpu_manager.go:414] "containerMap: removing container" podUID="aaf1c38e-171d-4874-8a38-9f0e03deacce" containerName="payload-pre-processing"
Oct 30 19:42:08.669854 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.669425    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaf1c38e-171d-4874-8a38-9f0e03deacce" containerName="payload-pre-processing"
Oct 30 19:42:08.669854 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.669544    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="b34f07ce-6edf-4493-b349-ad7916bb93aa" containerName="istio-proxy"
Oct 30 19:42:08.669854 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.669557    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="b34f07ce-6edf-4493-b349-ad7916bb93aa" containerName="istio-proxy"
Oct 30 19:42:08.669854 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.669566    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="cffde88e-6bb5-4c7c-b408-bbf25e17f381" containerName="istio-proxy"
Oct 30 19:42:08.669854 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.669573    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="28d04237-1b82-4642-b811-5bd3cd6e2068" containerName="payload-processing"
Oct 30 19:42:08.669854 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.669579    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaf1c38e-171d-4874-8a38-9f0e03deacce" containerName="payload-pre-processing"
Oct 30 19:42:08.673359 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.673327    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b623ca-j9hhm"
Oct 30 19:42:08.685298 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.682497    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:42:08.685298 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.682574    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:42:08.685298 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.682651    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:42:08.685298 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.684351    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:42:08.689316 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.689140    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b623ca-j9hhm"]
Oct 30 19:42:08.795759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.795487    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c8baf37-eb48-49c1-b373-c358ae9ae548-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-b623ca-j9hhm\" (UID: \"8c8baf37-eb48-49c1-b373-c358ae9ae548\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b623ca-j9hhm"
Oct 30 19:42:08.795759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.795607    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c8baf37-eb48-49c1-b373-c358ae9ae548-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-b623ca-j9hhm\" (UID: \"8c8baf37-eb48-49c1-b373-c358ae9ae548\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b623ca-j9hhm"
Oct 30 19:42:08.795759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.795722    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsf4n\" (UniqueName: \"kubernetes.io/projected/8c8baf37-eb48-49c1-b373-c358ae9ae548-kube-api-access-gsf4n\") pod \"maas-api-revoke-keys-e2e-worker-w2-b623ca-j9hhm\" (UID: \"8c8baf37-eb48-49c1-b373-c358ae9ae548\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b623ca-j9hhm"
Oct 30 19:42:08.813104 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.813067    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt"]
Oct 30 19:42:08.819895 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.819855    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ca9d14-openshift-default-65f96b67bf-klhmt"]
Oct 30 19:42:08.907003 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.906964    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c8baf37-eb48-49c1-b373-c358ae9ae548-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-b623ca-j9hhm\" (UID: \"8c8baf37-eb48-49c1-b373-c358ae9ae548\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b623ca-j9hhm"
Oct 30 19:42:08.907223 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.907067    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c8baf37-eb48-49c1-b373-c358ae9ae548-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-b623ca-j9hhm\" (UID: \"8c8baf37-eb48-49c1-b373-c358ae9ae548\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b623ca-j9hhm"
Oct 30 19:42:08.907223 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.907188    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsf4n\" (UniqueName: \"kubernetes.io/projected/8c8baf37-eb48-49c1-b373-c358ae9ae548-kube-api-access-gsf4n\") pod \"maas-api-revoke-keys-e2e-worker-w2-b623ca-j9hhm\" (UID: \"8c8baf37-eb48-49c1-b373-c358ae9ae548\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b623ca-j9hhm"
Oct 30 19:42:08.907674 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.907653    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c8baf37-eb48-49c1-b373-c358ae9ae548-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-b623ca-j9hhm\" (UID: \"8c8baf37-eb48-49c1-b373-c358ae9ae548\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b623ca-j9hhm"
Oct 30 19:42:08.909906 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.909881    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c8baf37-eb48-49c1-b373-c358ae9ae548-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-b623ca-j9hhm\" (UID: \"8c8baf37-eb48-49c1-b373-c358ae9ae548\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b623ca-j9hhm"
Oct 30 19:42:08.915660 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.915636    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsf4n\" (UniqueName: \"kubernetes.io/projected/8c8baf37-eb48-49c1-b373-c358ae9ae548-kube-api-access-gsf4n\") pod \"maas-api-revoke-keys-e2e-worker-w2-b623ca-j9hhm\" (UID: \"8c8baf37-eb48-49c1-b373-c358ae9ae548\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b623ca-j9hhm"
Oct 30 19:42:08.993958 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:08.993920    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b623ca-j9hhm"
Oct 30 19:42:09.297187 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:42:09.297150    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c8baf37_eb48_49c1_b373_c358ae9ae548.slice/crio-c4a7d41bd82546ac13dbd9ff420ea0f24290a1256363804a95c65c8441f1a891 WatchSource:0}: Error finding container c4a7d41bd82546ac13dbd9ff420ea0f24290a1256363804a95c65c8441f1a891: Status 404 returned error can't find the container with id c4a7d41bd82546ac13dbd9ff420ea0f24290a1256363804a95c65c8441f1a891
Oct 30 19:42:09.297187 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:09.297173    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b623ca-j9hhm"]
Oct 30 19:42:09.465060 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:09.465020    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b623ca-j9hhm" event={"ID":"8c8baf37-eb48-49c1-b373-c358ae9ae548","Type":"ContainerStarted","Data":"c4a7d41bd82546ac13dbd9ff420ea0f24290a1256363804a95c65c8441f1a891"}
Oct 30 19:42:10.477979 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:10.477931    5043 generic.go:356] "Generic (PLEG): container finished" podID="8c8baf37-eb48-49c1-b373-c358ae9ae548" containerID="38b56a8722633fab4e8704f17df2dd481c263aa9888bb9585fbba92601b20ef1" exitCode=0
Oct 30 19:42:10.478463 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:10.478077    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b623ca-j9hhm" event={"ID":"8c8baf37-eb48-49c1-b373-c358ae9ae548","Type":"ContainerDied","Data":"38b56a8722633fab4e8704f17df2dd481c263aa9888bb9585fbba92601b20ef1"}
Oct 30 19:42:10.650177 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:10.650080    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cffde88e-6bb5-4c7c-b408-bbf25e17f381" path="/var/lib/kubelet/pods/cffde88e-6bb5-4c7c-b408-bbf25e17f381/volumes"
Oct 30 19:42:11.680313 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:11.680286    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b623ca-j9hhm"
Oct 30 19:42:11.849773 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:11.849655    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c8baf37-eb48-49c1-b373-c358ae9ae548-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c8baf37-eb48-49c1-b373-c358ae9ae548-maas-api-openshift-ca\") pod \"8c8baf37-eb48-49c1-b373-c358ae9ae548\" (UID: \"8c8baf37-eb48-49c1-b373-c358ae9ae548\") "
Oct 30 19:42:11.849941 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:11.849826    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c8baf37-eb48-49c1-b373-c358ae9ae548-kube-api-access-gsf4n\" (UniqueName: \"kubernetes.io/projected/8c8baf37-eb48-49c1-b373-c358ae9ae548-kube-api-access-gsf4n\") pod \"8c8baf37-eb48-49c1-b373-c358ae9ae548\" (UID: \"8c8baf37-eb48-49c1-b373-c358ae9ae548\") "
Oct 30 19:42:11.849941 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:11.849924    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c8baf37-eb48-49c1-b373-c358ae9ae548-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c8baf37-eb48-49c1-b373-c358ae9ae548-maas-api-xks-ca\") pod \"8c8baf37-eb48-49c1-b373-c358ae9ae548\" (UID: \"8c8baf37-eb48-49c1-b373-c358ae9ae548\") "
Oct 30 19:42:11.850058 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:11.850014    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c8baf37-eb48-49c1-b373-c358ae9ae548-maas-api-openshift-ca" pod "8c8baf37-eb48-49c1-b373-c358ae9ae548" (UID: "8c8baf37-eb48-49c1-b373-c358ae9ae548"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:42:11.850564 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:11.850545    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c8baf37-eb48-49c1-b373-c358ae9ae548-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:11.852143 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:11.852117    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c8baf37-eb48-49c1-b373-c358ae9ae548-maas-api-xks-ca" pod "8c8baf37-eb48-49c1-b373-c358ae9ae548" (UID: "8c8baf37-eb48-49c1-b373-c358ae9ae548"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:42:11.852256 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:11.852160    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c8baf37-eb48-49c1-b373-c358ae9ae548-kube-api-access-gsf4n" pod "8c8baf37-eb48-49c1-b373-c358ae9ae548" (UID: "8c8baf37-eb48-49c1-b373-c358ae9ae548"). InnerVolumeSpecName "kube-api-access-gsf4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:11.951568 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:11.951540    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsf4n\" (UniqueName: \"kubernetes.io/projected/8c8baf37-eb48-49c1-b373-c358ae9ae548-kube-api-access-gsf4n\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:11.951568 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:11.951567    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c8baf37-eb48-49c1-b373-c358ae9ae548-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:12.505083 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:12.505054    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b623ca-j9hhm"
Oct 30 19:42:12.505281 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:12.505096    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b623ca-j9hhm" event={"ID":"8c8baf37-eb48-49c1-b373-c358ae9ae548","Type":"ContainerDied","Data":"c4a7d41bd82546ac13dbd9ff420ea0f24290a1256363804a95c65c8441f1a891"}
Oct 30 19:42:12.505281 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:12.505147    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4a7d41bd82546ac13dbd9ff420ea0f24290a1256363804a95c65c8441f1a891"
Oct 30 19:42:13.394828 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:13.394784    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"]
Oct 30 19:42:13.407329 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:13.407292    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"]
Oct 30 19:42:24.425477 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:24.425445    5043 scope.go:122] "RemoveContainer" containerID="64db22f423648649dd1d2d8b617043e30d9f6c4c17d9050cd98c4f6538405494"
Oct 30 19:42:24.437892 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:24.437871    5043 scope.go:122] "RemoveContainer" containerID="4bdc9a97d9d5f4e1494f5a2d3f8b3ddd4fc4987c140a304072d2a67ced30fb13"
Oct 30 19:42:24.450305 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:24.450280    5043 scope.go:122] "RemoveContainer" containerID="ea5c03c777d8a153bfb8f787cfd04b1610ed1c2c1a4c658e83f1ebbdeddc3751"
Oct 30 19:42:24.980498 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:42:24.980456    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:42:27.567775 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:27.567699    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b489b261-cc58d"]
Oct 30 19:42:27.568641 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:27.568616    5043 cpu_manager.go:414] "containerMap: removing container" podUID="8c8baf37-eb48-49c1-b373-c358ae9ae548" containerName="revoke-keys"
Oct 30 19:42:27.568798 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:27.568642    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c8baf37-eb48-49c1-b373-c358ae9ae548" containerName="revoke-keys"
Oct 30 19:42:27.568861 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:27.568841    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c8baf37-eb48-49c1-b373-c358ae9ae548" containerName="revoke-keys"
Oct 30 19:42:27.574796 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:27.574720    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b489b261-cc58d"
Oct 30 19:42:27.581359 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:27.581298    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:42:27.581584 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:27.581450    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:42:27.581584 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:27.581479    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:42:27.581584 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:27.581535    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:42:27.583215 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:27.583192    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbb1760d-v497b"]
Oct 30 19:42:27.588277 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:27.588255    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb1760d-v497b"
Oct 30 19:42:27.592783 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:27.588576    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b489b261-cc58d"]
Oct 30 19:42:27.605993 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:27.605957    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbb1760d-v497b"]
Oct 30 19:42:27.645980 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:27.645941    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgbs2\" (UniqueName: \"kubernetes.io/projected/a2d3d64b-2e22-491b-a612-078423f87212-kube-api-access-hgbs2\") pod \"maas-api-revoke-sub-bbb1760d-v497b\" (UID: \"a2d3d64b-2e22-491b-a612-078423f87212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb1760d-v497b"
Oct 30 19:42:27.646204 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:27.646044    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2d3d64b-2e22-491b-a612-078423f87212-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbb1760d-v497b\" (UID: \"a2d3d64b-2e22-491b-a612-078423f87212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb1760d-v497b"
Oct 30 19:42:27.646204 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:27.646114    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2d3d64b-2e22-491b-a612-078423f87212-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbb1760d-v497b\" (UID: \"a2d3d64b-2e22-491b-a612-078423f87212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb1760d-v497b"
Oct 30 19:42:27.646204 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:27.646186    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljxfd\" (UniqueName: \"kubernetes.io/projected/448520e8-011d-4079-b0b2-ddb5263fd382-kube-api-access-ljxfd\") pod \"maas-api-revoke-sub-b489b261-cc58d\" (UID: \"448520e8-011d-4079-b0b2-ddb5263fd382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b489b261-cc58d"
Oct 30 19:42:27.646446 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:27.646266    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/448520e8-011d-4079-b0b2-ddb5263fd382-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b489b261-cc58d\" (UID: \"448520e8-011d-4079-b0b2-ddb5263fd382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b489b261-cc58d"
Oct 30 19:42:27.646446 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:27.646326    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/448520e8-011d-4079-b0b2-ddb5263fd382-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b489b261-cc58d\" (UID: \"448520e8-011d-4079-b0b2-ddb5263fd382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b489b261-cc58d"
Oct 30 19:42:27.748020 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:27.747982    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/448520e8-011d-4079-b0b2-ddb5263fd382-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b489b261-cc58d\" (UID: \"448520e8-011d-4079-b0b2-ddb5263fd382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b489b261-cc58d"
Oct 30 19:42:27.748187 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:27.748043    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/448520e8-011d-4079-b0b2-ddb5263fd382-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b489b261-cc58d\" (UID: \"448520e8-011d-4079-b0b2-ddb5263fd382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b489b261-cc58d"
Oct 30 19:42:27.748187 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:27.748131    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgbs2\" (UniqueName: \"kubernetes.io/projected/a2d3d64b-2e22-491b-a612-078423f87212-kube-api-access-hgbs2\") pod \"maas-api-revoke-sub-bbb1760d-v497b\" (UID: \"a2d3d64b-2e22-491b-a612-078423f87212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb1760d-v497b"
Oct 30 19:42:27.748187 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:27.748185    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2d3d64b-2e22-491b-a612-078423f87212-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbb1760d-v497b\" (UID: \"a2d3d64b-2e22-491b-a612-078423f87212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb1760d-v497b"
Oct 30 19:42:27.748454 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:27.748428    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2d3d64b-2e22-491b-a612-078423f87212-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbb1760d-v497b\" (UID: \"a2d3d64b-2e22-491b-a612-078423f87212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb1760d-v497b"
Oct 30 19:42:27.748562 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:27.748542    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljxfd\" (UniqueName: \"kubernetes.io/projected/448520e8-011d-4079-b0b2-ddb5263fd382-kube-api-access-ljxfd\") pod \"maas-api-revoke-sub-b489b261-cc58d\" (UID: \"448520e8-011d-4079-b0b2-ddb5263fd382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b489b261-cc58d"
Oct 30 19:42:27.748956 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:27.748929    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/448520e8-011d-4079-b0b2-ddb5263fd382-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b489b261-cc58d\" (UID: \"448520e8-011d-4079-b0b2-ddb5263fd382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b489b261-cc58d"
Oct 30 19:42:27.749249 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:27.749070    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2d3d64b-2e22-491b-a612-078423f87212-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbb1760d-v497b\" (UID: \"a2d3d64b-2e22-491b-a612-078423f87212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb1760d-v497b"
Oct 30 19:42:27.750924 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:27.750900    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2d3d64b-2e22-491b-a612-078423f87212-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbb1760d-v497b\" (UID: \"a2d3d64b-2e22-491b-a612-078423f87212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb1760d-v497b"
Oct 30 19:42:27.751028 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:27.750932    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/448520e8-011d-4079-b0b2-ddb5263fd382-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b489b261-cc58d\" (UID: \"448520e8-011d-4079-b0b2-ddb5263fd382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b489b261-cc58d"
Oct 30 19:42:27.759224 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:27.759188    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljxfd\" (UniqueName: \"kubernetes.io/projected/448520e8-011d-4079-b0b2-ddb5263fd382-kube-api-access-ljxfd\") pod \"maas-api-revoke-sub-b489b261-cc58d\" (UID: \"448520e8-011d-4079-b0b2-ddb5263fd382\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b489b261-cc58d"
Oct 30 19:42:27.760366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:27.760340    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgbs2\" (UniqueName: \"kubernetes.io/projected/a2d3d64b-2e22-491b-a612-078423f87212-kube-api-access-hgbs2\") pod \"maas-api-revoke-sub-bbb1760d-v497b\" (UID: \"a2d3d64b-2e22-491b-a612-078423f87212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb1760d-v497b"
Oct 30 19:42:27.896456 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:27.896356    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b489b261-cc58d"
Oct 30 19:42:27.921095 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:27.921063    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb1760d-v497b"
Oct 30 19:42:28.218976 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:28.218944    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b489b261-cc58d"]
Oct 30 19:42:28.255399 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:28.255366    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbb1760d-v497b"]
Oct 30 19:42:28.255635 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:42:28.255608    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2d3d64b_2e22_491b_a612_078423f87212.slice/crio-ae647b79909137007e1807fdb9fefa331390bae77afe345f5e4ccade5e818f73 WatchSource:0}: Error finding container ae647b79909137007e1807fdb9fefa331390bae77afe345f5e4ccade5e818f73: Status 404 returned error can't find the container with id ae647b79909137007e1807fdb9fefa331390bae77afe345f5e4ccade5e818f73
Oct 30 19:42:28.420876 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:28.420834    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" containerID="cri-o://40f0dca84100a818eec5a1f5fe6451dd70e441f3405c85449d8d4942b7b8ec03" gracePeriod=45
Oct 30 19:42:28.431262 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:28.431188    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main" containerID="cri-o://f2c51ed688b58c5c7e3441d6efd51e87d05bc96c9efe5e573a313573bfda7348" gracePeriod=45
Oct 30 19:42:28.511446 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:28.511403    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": dial tcp 10.132.0.64:8000: connect: connection refused"
Oct 30 19:42:28.663781 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:28.663706    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b489b261-cc58d" event={"ID":"448520e8-011d-4079-b0b2-ddb5263fd382","Type":"ContainerStarted","Data":"47c9610e63fb9d423ff00c5d2af2df945e68cc5418ed9bd032de110c6e9e3cdb"}
Oct 30 19:42:28.665483 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:28.665317    5043 generic.go:356] "Generic (PLEG): container finished" podID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerID="f2c51ed688b58c5c7e3441d6efd51e87d05bc96c9efe5e573a313573bfda7348" exitCode=0
Oct 30 19:42:28.665483 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:28.665444    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" event={"ID":"27af3d5f-b68f-4b68-aecf-2fe5f82b4891","Type":"ContainerDied","Data":"f2c51ed688b58c5c7e3441d6efd51e87d05bc96c9efe5e573a313573bfda7348"}
Oct 30 19:42:28.667621 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:28.667597    5043 generic.go:356] "Generic (PLEG): container finished" podID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerID="40f0dca84100a818eec5a1f5fe6451dd70e441f3405c85449d8d4942b7b8ec03" exitCode=0
Oct 30 19:42:28.667820 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:28.667663    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" event={"ID":"39590583-81fd-4e35-9047-ebd7a47b4fce","Type":"ContainerDied","Data":"40f0dca84100a818eec5a1f5fe6451dd70e441f3405c85449d8d4942b7b8ec03"}
Oct 30 19:42:28.669855 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:28.669024    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb1760d-v497b" event={"ID":"a2d3d64b-2e22-491b-a612-078423f87212","Type":"ContainerStarted","Data":"ae647b79909137007e1807fdb9fefa331390bae77afe345f5e4ccade5e818f73"}
Oct 30 19:42:28.827263 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:28.827232    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:42:28.901496 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:28.901459    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:42:28.910659 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:28.910633    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27af3d5f-b68f-4b68-aecf-2fe5f82b4891-dshm\" (UniqueName: \"kubernetes.io/empty-dir/27af3d5f-b68f-4b68-aecf-2fe5f82b4891-dshm\") pod \"27af3d5f-b68f-4b68-aecf-2fe5f82b4891\" (UID: \"27af3d5f-b68f-4b68-aecf-2fe5f82b4891\") "
Oct 30 19:42:28.910659 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:28.910665    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27af3d5f-b68f-4b68-aecf-2fe5f82b4891-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27af3d5f-b68f-4b68-aecf-2fe5f82b4891-tmp-dir\") pod \"27af3d5f-b68f-4b68-aecf-2fe5f82b4891\" (UID: \"27af3d5f-b68f-4b68-aecf-2fe5f82b4891\") "
Oct 30 19:42:28.911015 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:28.910993    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27af3d5f-b68f-4b68-aecf-2fe5f82b4891-tmp-dir" pod "27af3d5f-b68f-4b68-aecf-2fe5f82b4891" (UID: "27af3d5f-b68f-4b68-aecf-2fe5f82b4891"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:42:28.911368 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:28.911339    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27af3d5f-b68f-4b68-aecf-2fe5f82b4891-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27af3d5f-b68f-4b68-aecf-2fe5f82b4891-model-cache\") pod \"27af3d5f-b68f-4b68-aecf-2fe5f82b4891\" (UID: \"27af3d5f-b68f-4b68-aecf-2fe5f82b4891\") "
Oct 30 19:42:28.911514 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:28.911376    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27af3d5f-b68f-4b68-aecf-2fe5f82b4891-tls-certs\" (UniqueName: \"kubernetes.io/secret/27af3d5f-b68f-4b68-aecf-2fe5f82b4891-tls-certs\") pod \"27af3d5f-b68f-4b68-aecf-2fe5f82b4891\" (UID: \"27af3d5f-b68f-4b68-aecf-2fe5f82b4891\") "
Oct 30 19:42:28.911795 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:28.911616    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27af3d5f-b68f-4b68-aecf-2fe5f82b4891-home\" (UniqueName: \"kubernetes.io/empty-dir/27af3d5f-b68f-4b68-aecf-2fe5f82b4891-home\") pod \"27af3d5f-b68f-4b68-aecf-2fe5f82b4891\" (UID: \"27af3d5f-b68f-4b68-aecf-2fe5f82b4891\") "
Oct 30 19:42:28.911795 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:28.911656    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27af3d5f-b68f-4b68-aecf-2fe5f82b4891-kube-api-access-lr4kh\" (UniqueName: \"kubernetes.io/projected/27af3d5f-b68f-4b68-aecf-2fe5f82b4891-kube-api-access-lr4kh\") pod \"27af3d5f-b68f-4b68-aecf-2fe5f82b4891\" (UID: \"27af3d5f-b68f-4b68-aecf-2fe5f82b4891\") "
Oct 30 19:42:28.912138 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:28.912120    5043 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27af3d5f-b68f-4b68-aecf-2fe5f82b4891-tmp-dir\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:28.912701 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:28.912672    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27af3d5f-b68f-4b68-aecf-2fe5f82b4891-home" pod "27af3d5f-b68f-4b68-aecf-2fe5f82b4891" (UID: "27af3d5f-b68f-4b68-aecf-2fe5f82b4891"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:42:28.912865 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:28.912814    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27af3d5f-b68f-4b68-aecf-2fe5f82b4891-model-cache" pod "27af3d5f-b68f-4b68-aecf-2fe5f82b4891" (UID: "27af3d5f-b68f-4b68-aecf-2fe5f82b4891"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:42:28.913509 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:28.913483    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27af3d5f-b68f-4b68-aecf-2fe5f82b4891-dshm" pod "27af3d5f-b68f-4b68-aecf-2fe5f82b4891" (UID: "27af3d5f-b68f-4b68-aecf-2fe5f82b4891"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:42:28.914367 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:28.914339    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27af3d5f-b68f-4b68-aecf-2fe5f82b4891-tls-certs" pod "27af3d5f-b68f-4b68-aecf-2fe5f82b4891" (UID: "27af3d5f-b68f-4b68-aecf-2fe5f82b4891"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:42:28.914813 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:28.914786    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27af3d5f-b68f-4b68-aecf-2fe5f82b4891-kube-api-access-lr4kh" pod "27af3d5f-b68f-4b68-aecf-2fe5f82b4891" (UID: "27af3d5f-b68f-4b68-aecf-2fe5f82b4891"). InnerVolumeSpecName "kube-api-access-lr4kh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:29.013544 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:29.013510    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39590583-81fd-4e35-9047-ebd7a47b4fce-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39590583-81fd-4e35-9047-ebd7a47b4fce-tmp-dir\") pod \"39590583-81fd-4e35-9047-ebd7a47b4fce\" (UID: \"39590583-81fd-4e35-9047-ebd7a47b4fce\") "
Oct 30 19:42:29.013757 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:29.013565    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39590583-81fd-4e35-9047-ebd7a47b4fce-home\" (UniqueName: \"kubernetes.io/empty-dir/39590583-81fd-4e35-9047-ebd7a47b4fce-home\") pod \"39590583-81fd-4e35-9047-ebd7a47b4fce\" (UID: \"39590583-81fd-4e35-9047-ebd7a47b4fce\") "
Oct 30 19:42:29.013757 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:29.013700    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39590583-81fd-4e35-9047-ebd7a47b4fce-dshm\" (UniqueName: \"kubernetes.io/empty-dir/39590583-81fd-4e35-9047-ebd7a47b4fce-dshm\") pod \"39590583-81fd-4e35-9047-ebd7a47b4fce\" (UID: \"39590583-81fd-4e35-9047-ebd7a47b4fce\") "
Oct 30 19:42:29.013757 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:29.013752    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39590583-81fd-4e35-9047-ebd7a47b4fce-tls-certs\" (UniqueName: \"kubernetes.io/secret/39590583-81fd-4e35-9047-ebd7a47b4fce-tls-certs\") pod \"39590583-81fd-4e35-9047-ebd7a47b4fce\" (UID: \"39590583-81fd-4e35-9047-ebd7a47b4fce\") "
Oct 30 19:42:29.013969 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:29.013827    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39590583-81fd-4e35-9047-ebd7a47b4fce-tmp-dir" pod "39590583-81fd-4e35-9047-ebd7a47b4fce" (UID: "39590583-81fd-4e35-9047-ebd7a47b4fce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:42:29.013969 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:29.013906    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39590583-81fd-4e35-9047-ebd7a47b4fce-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39590583-81fd-4e35-9047-ebd7a47b4fce-model-cache\") pod \"39590583-81fd-4e35-9047-ebd7a47b4fce\" (UID: \"39590583-81fd-4e35-9047-ebd7a47b4fce\") "
Oct 30 19:42:29.014087 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:29.013986    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39590583-81fd-4e35-9047-ebd7a47b4fce-home" pod "39590583-81fd-4e35-9047-ebd7a47b4fce" (UID: "39590583-81fd-4e35-9047-ebd7a47b4fce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:42:29.014087 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:29.014002    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39590583-81fd-4e35-9047-ebd7a47b4fce-kube-api-access-j7twl\" (UniqueName: \"kubernetes.io/projected/39590583-81fd-4e35-9047-ebd7a47b4fce-kube-api-access-j7twl\") pod \"39590583-81fd-4e35-9047-ebd7a47b4fce\" (UID: \"39590583-81fd-4e35-9047-ebd7a47b4fce\") "
Oct 30 19:42:29.014199 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:29.014160    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39590583-81fd-4e35-9047-ebd7a47b4fce-model-cache" pod "39590583-81fd-4e35-9047-ebd7a47b4fce" (UID: "39590583-81fd-4e35-9047-ebd7a47b4fce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:42:29.014696 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:29.014675    5043 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39590583-81fd-4e35-9047-ebd7a47b4fce-model-cache\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:29.014848 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:29.014702    5043 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39590583-81fd-4e35-9047-ebd7a47b4fce-tmp-dir\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:29.014848 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:29.014715    5043 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27af3d5f-b68f-4b68-aecf-2fe5f82b4891-home\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:29.014848 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:29.014726    5043 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39590583-81fd-4e35-9047-ebd7a47b4fce-home\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:29.014848 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:29.014756    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr4kh\" (UniqueName: \"kubernetes.io/projected/27af3d5f-b68f-4b68-aecf-2fe5f82b4891-kube-api-access-lr4kh\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:29.014848 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:29.014768    5043 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27af3d5f-b68f-4b68-aecf-2fe5f82b4891-dshm\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:29.014848 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:29.014781    5043 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27af3d5f-b68f-4b68-aecf-2fe5f82b4891-model-cache\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:29.014848 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:29.014794    5043 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27af3d5f-b68f-4b68-aecf-2fe5f82b4891-tls-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:29.016140 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:29.016113    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39590583-81fd-4e35-9047-ebd7a47b4fce-tls-certs" pod "39590583-81fd-4e35-9047-ebd7a47b4fce" (UID: "39590583-81fd-4e35-9047-ebd7a47b4fce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:42:29.016140 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:29.016114    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39590583-81fd-4e35-9047-ebd7a47b4fce-kube-api-access-j7twl" pod "39590583-81fd-4e35-9047-ebd7a47b4fce" (UID: "39590583-81fd-4e35-9047-ebd7a47b4fce"). InnerVolumeSpecName "kube-api-access-j7twl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:29.016283 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:29.016264    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39590583-81fd-4e35-9047-ebd7a47b4fce-dshm" pod "39590583-81fd-4e35-9047-ebd7a47b4fce" (UID: "39590583-81fd-4e35-9047-ebd7a47b4fce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:42:29.116305 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:29.116207    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7twl\" (UniqueName: \"kubernetes.io/projected/39590583-81fd-4e35-9047-ebd7a47b4fce-kube-api-access-j7twl\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:29.116305 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:29.116255    5043 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39590583-81fd-4e35-9047-ebd7a47b4fce-dshm\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:29.116305 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:29.116271    5043 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39590583-81fd-4e35-9047-ebd7a47b4fce-tls-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:29.687120 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:29.687036    5043 generic.go:356] "Generic (PLEG): container finished" podID="448520e8-011d-4079-b0b2-ddb5263fd382" containerID="61d83a1f308297eb84a8f91542319f30c10099fdbd43e18617e00df725e99527" exitCode=0
Oct 30 19:42:29.687582 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:29.687129    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b489b261-cc58d" event={"ID":"448520e8-011d-4079-b0b2-ddb5263fd382","Type":"ContainerDied","Data":"61d83a1f308297eb84a8f91542319f30c10099fdbd43e18617e00df725e99527"}
Oct 30 19:42:29.688750 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:29.688719    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"
Oct 30 19:42:29.688872 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:29.688763    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs" event={"ID":"27af3d5f-b68f-4b68-aecf-2fe5f82b4891","Type":"ContainerDied","Data":"b00e81026f63eaeaa56b453d03700d243e16a1cd718ce521adee5f380cebfa52"}
Oct 30 19:42:29.688872 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:29.688807    5043 scope.go:122] "RemoveContainer" containerID="f2c51ed688b58c5c7e3441d6efd51e87d05bc96c9efe5e573a313573bfda7348"
Oct 30 19:42:29.692001 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:29.690582    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg" event={"ID":"39590583-81fd-4e35-9047-ebd7a47b4fce","Type":"ContainerDied","Data":"cbcaa5a305e8d60dbd249f5adb24893e54d9b9afdfa13f7b35424a387fa3969c"}
Oct 30 19:42:29.692001 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:29.690587    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"
Oct 30 19:42:29.697564 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:29.697533    5043 generic.go:356] "Generic (PLEG): container finished" podID="a2d3d64b-2e22-491b-a612-078423f87212" containerID="3e6cc256fc8e08ee6ddfbc8f09156772efcd2374dc23a2107e8114f45b12b9b4" exitCode=0
Oct 30 19:42:29.697684 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:29.697640    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb1760d-v497b" event={"ID":"a2d3d64b-2e22-491b-a612-078423f87212","Type":"ContainerDied","Data":"3e6cc256fc8e08ee6ddfbc8f09156772efcd2374dc23a2107e8114f45b12b9b4"}
Oct 30 19:42:29.703804 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:29.703784    5043 scope.go:122] "RemoveContainer" containerID="40f0dca84100a818eec5a1f5fe6451dd70e441f3405c85449d8d4942b7b8ec03"
Oct 30 19:42:29.728112 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:29.728077    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"]
Oct 30 19:42:29.731916 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:29.731891    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-b623ca/premium-w2-b623ca-kserve-55dcdc76cc-pjqrg"]
Oct 30 19:42:29.770114 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:29.770076    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"]
Oct 30 19:42:29.779387 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:29.779355    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-b623ca/simulator-w2-b623ca-kserve-57bf844dd4-d67rs"]
Oct 30 19:42:30.649435 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:30.649344    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" path="/var/lib/kubelet/pods/27af3d5f-b68f-4b68-aecf-2fe5f82b4891/volumes"
Oct 30 19:42:30.650089 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:30.650070    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" path="/var/lib/kubelet/pods/39590583-81fd-4e35-9047-ebd7a47b4fce/volumes"
Oct 30 19:42:30.915100 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:30.915079    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b489b261-cc58d"
Oct 30 19:42:30.948243 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:30.948216    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb1760d-v497b"
Oct 30 19:42:31.046283 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:31.046249    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/448520e8-011d-4079-b0b2-ddb5263fd382-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/448520e8-011d-4079-b0b2-ddb5263fd382-maas-api-xks-ca\") pod \"448520e8-011d-4079-b0b2-ddb5263fd382\" (UID: \"448520e8-011d-4079-b0b2-ddb5263fd382\") "
Oct 30 19:42:31.046528 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:31.046359    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2d3d64b-2e22-491b-a612-078423f87212-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2d3d64b-2e22-491b-a612-078423f87212-maas-api-openshift-ca\") pod \"a2d3d64b-2e22-491b-a612-078423f87212\" (UID: \"a2d3d64b-2e22-491b-a612-078423f87212\") "
Oct 30 19:42:31.046528 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:31.046429    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/448520e8-011d-4079-b0b2-ddb5263fd382-kube-api-access-ljxfd\" (UniqueName: \"kubernetes.io/projected/448520e8-011d-4079-b0b2-ddb5263fd382-kube-api-access-ljxfd\") pod \"448520e8-011d-4079-b0b2-ddb5263fd382\" (UID: \"448520e8-011d-4079-b0b2-ddb5263fd382\") "
Oct 30 19:42:31.046626 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:31.046565    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/448520e8-011d-4079-b0b2-ddb5263fd382-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/448520e8-011d-4079-b0b2-ddb5263fd382-maas-api-openshift-ca\") pod \"448520e8-011d-4079-b0b2-ddb5263fd382\" (UID: \"448520e8-011d-4079-b0b2-ddb5263fd382\") "
Oct 30 19:42:31.046930 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:31.046902    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2d3d64b-2e22-491b-a612-078423f87212-kube-api-access-hgbs2\" (UniqueName: \"kubernetes.io/projected/a2d3d64b-2e22-491b-a612-078423f87212-kube-api-access-hgbs2\") pod \"a2d3d64b-2e22-491b-a612-078423f87212\" (UID: \"a2d3d64b-2e22-491b-a612-078423f87212\") "
Oct 30 19:42:31.047297 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:31.046982    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2d3d64b-2e22-491b-a612-078423f87212-maas-api-openshift-ca" pod "a2d3d64b-2e22-491b-a612-078423f87212" (UID: "a2d3d64b-2e22-491b-a612-078423f87212"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:42:31.047297 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:31.046949    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2d3d64b-2e22-491b-a612-078423f87212-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2d3d64b-2e22-491b-a612-078423f87212-maas-api-xks-ca\") pod \"a2d3d64b-2e22-491b-a612-078423f87212\" (UID: \"a2d3d64b-2e22-491b-a612-078423f87212\") "
Oct 30 19:42:31.048317 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:31.048285    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2d3d64b-2e22-491b-a612-078423f87212-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:31.048937 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:31.048909    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/448520e8-011d-4079-b0b2-ddb5263fd382-maas-api-openshift-ca" pod "448520e8-011d-4079-b0b2-ddb5263fd382" (UID: "448520e8-011d-4079-b0b2-ddb5263fd382"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:42:31.049565 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:31.049482    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/448520e8-011d-4079-b0b2-ddb5263fd382-maas-api-xks-ca" pod "448520e8-011d-4079-b0b2-ddb5263fd382" (UID: "448520e8-011d-4079-b0b2-ddb5263fd382"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:42:31.052076 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:31.052047    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2d3d64b-2e22-491b-a612-078423f87212-kube-api-access-hgbs2" pod "a2d3d64b-2e22-491b-a612-078423f87212" (UID: "a2d3d64b-2e22-491b-a612-078423f87212"). InnerVolumeSpecName "kube-api-access-hgbs2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:31.052076 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:31.052058    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2d3d64b-2e22-491b-a612-078423f87212-maas-api-xks-ca" pod "a2d3d64b-2e22-491b-a612-078423f87212" (UID: "a2d3d64b-2e22-491b-a612-078423f87212"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:42:31.052282 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:31.052259    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/448520e8-011d-4079-b0b2-ddb5263fd382-kube-api-access-ljxfd" pod "448520e8-011d-4079-b0b2-ddb5263fd382" (UID: "448520e8-011d-4079-b0b2-ddb5263fd382"). InnerVolumeSpecName "kube-api-access-ljxfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:31.149746 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:31.149709    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/448520e8-011d-4079-b0b2-ddb5263fd382-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:31.149923 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:31.149778    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljxfd\" (UniqueName: \"kubernetes.io/projected/448520e8-011d-4079-b0b2-ddb5263fd382-kube-api-access-ljxfd\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:31.149923 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:31.149789    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/448520e8-011d-4079-b0b2-ddb5263fd382-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:31.149923 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:31.149797    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgbs2\" (UniqueName: \"kubernetes.io/projected/a2d3d64b-2e22-491b-a612-078423f87212-kube-api-access-hgbs2\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:31.149923 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:31.149808    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2d3d64b-2e22-491b-a612-078423f87212-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:31.719219 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:31.719192    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b489b261-cc58d"
Oct 30 19:42:31.719378 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:31.719230    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b489b261-cc58d" event={"ID":"448520e8-011d-4079-b0b2-ddb5263fd382","Type":"ContainerDied","Data":"47c9610e63fb9d423ff00c5d2af2df945e68cc5418ed9bd032de110c6e9e3cdb"}
Oct 30 19:42:31.719378 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:31.719314    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47c9610e63fb9d423ff00c5d2af2df945e68cc5418ed9bd032de110c6e9e3cdb"
Oct 30 19:42:31.720982 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:31.720958    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb1760d-v497b"
Oct 30 19:42:31.721105 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:31.720996    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbb1760d-v497b" event={"ID":"a2d3d64b-2e22-491b-a612-078423f87212","Type":"ContainerDied","Data":"ae647b79909137007e1807fdb9fefa331390bae77afe345f5e4ccade5e818f73"}
Oct 30 19:42:31.721105 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:31.721037    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae647b79909137007e1807fdb9fefa331390bae77afe345f5e4ccade5e818f73"
Oct 30 19:42:38.443363 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:38.443320    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq"]
Oct 30 19:42:38.444309 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:38.444286    5043 cpu_manager.go:414] "containerMap: removing container" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main"
Oct 30 19:42:38.444309 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:38.444309    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main"
Oct 30 19:42:38.444477 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:38.444322    5043 cpu_manager.go:414] "containerMap: removing container" podUID="a2d3d64b-2e22-491b-a612-078423f87212" containerName="revoke-keys"
Oct 30 19:42:38.444477 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:38.444330    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2d3d64b-2e22-491b-a612-078423f87212" containerName="revoke-keys"
Oct 30 19:42:38.444477 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:38.444376    5043 cpu_manager.go:414] "containerMap: removing container" podUID="448520e8-011d-4079-b0b2-ddb5263fd382" containerName="revoke-keys"
Oct 30 19:42:38.444477 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:38.444385    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="448520e8-011d-4079-b0b2-ddb5263fd382" containerName="revoke-keys"
Oct 30 19:42:38.444477 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:38.444412    5043 cpu_manager.go:414] "containerMap: removing container" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main"
Oct 30 19:42:38.444477 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:38.444418    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main"
Oct 30 19:42:38.444816 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:38.444627    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="27af3d5f-b68f-4b68-aecf-2fe5f82b4891" containerName="main"
Oct 30 19:42:38.444816 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:38.444642    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2d3d64b-2e22-491b-a612-078423f87212" containerName="revoke-keys"
Oct 30 19:42:38.444816 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:38.444661    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="448520e8-011d-4079-b0b2-ddb5263fd382" containerName="revoke-keys"
Oct 30 19:42:38.444816 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:38.444675    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="39590583-81fd-4e35-9047-ebd7a47b4fce" containerName="main"
Oct 30 19:42:38.448858 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:38.448835    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq"
Oct 30 19:42:38.456881 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:38.456852    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-3e35df-openshift-default-dockercfg-5snvf\""
Oct 30 19:42:38.463488 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:38.463453    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq"]
Oct 30 19:42:38.588421 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:38.588386    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk2sb\" (UniqueName: \"kubernetes.io/projected/3c89df4b-9274-4375-8ff8-47763bbf9182-kube-api-access-kk2sb\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq\" (UID: \"3c89df4b-9274-4375-8ff8-47763bbf9182\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq"
Oct 30 19:42:38.588608 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:38.588525    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c89df4b-9274-4375-8ff8-47763bbf9182-istio-data\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq\" (UID: \"3c89df4b-9274-4375-8ff8-47763bbf9182\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq"
Oct 30 19:42:38.588608 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:38.588572    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c89df4b-9274-4375-8ff8-47763bbf9182-istio-token\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq\" (UID: \"3c89df4b-9274-4375-8ff8-47763bbf9182\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq"
Oct 30 19:42:38.588726 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:38.588687    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c89df4b-9274-4375-8ff8-47763bbf9182-credential-socket\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq\" (UID: \"3c89df4b-9274-4375-8ff8-47763bbf9182\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq"
Oct 30 19:42:38.588823 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:38.588772    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c89df4b-9274-4375-8ff8-47763bbf9182-workload-certs\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq\" (UID: \"3c89df4b-9274-4375-8ff8-47763bbf9182\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq"
Oct 30 19:42:38.588874 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:38.588853    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c89df4b-9274-4375-8ff8-47763bbf9182-istiod-ca-cert\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq\" (UID: \"3c89df4b-9274-4375-8ff8-47763bbf9182\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq"
Oct 30 19:42:38.588907 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:38.588879    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c89df4b-9274-4375-8ff8-47763bbf9182-istio-podinfo\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq\" (UID: \"3c89df4b-9274-4375-8ff8-47763bbf9182\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq"
Oct 30 19:42:38.589028 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:38.588995    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c89df4b-9274-4375-8ff8-47763bbf9182-workload-socket\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq\" (UID: \"3c89df4b-9274-4375-8ff8-47763bbf9182\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq"
Oct 30 19:42:38.589093 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:38.589041    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c89df4b-9274-4375-8ff8-47763bbf9182-istio-envoy\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq\" (UID: \"3c89df4b-9274-4375-8ff8-47763bbf9182\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq"
Oct 30 19:42:38.698568 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:38.693058    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c89df4b-9274-4375-8ff8-47763bbf9182-istio-data\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq\" (UID: \"3c89df4b-9274-4375-8ff8-47763bbf9182\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq"
Oct 30 19:42:38.698568 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:38.693132    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c89df4b-9274-4375-8ff8-47763bbf9182-istio-token\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq\" (UID: \"3c89df4b-9274-4375-8ff8-47763bbf9182\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq"
Oct 30 19:42:38.698568 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:38.693220    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c89df4b-9274-4375-8ff8-47763bbf9182-credential-socket\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq\" (UID: \"3c89df4b-9274-4375-8ff8-47763bbf9182\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq"
Oct 30 19:42:38.698568 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:38.693283    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c89df4b-9274-4375-8ff8-47763bbf9182-workload-certs\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq\" (UID: \"3c89df4b-9274-4375-8ff8-47763bbf9182\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq"
Oct 30 19:42:38.698568 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:38.693354    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c89df4b-9274-4375-8ff8-47763bbf9182-istiod-ca-cert\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq\" (UID: \"3c89df4b-9274-4375-8ff8-47763bbf9182\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq"
Oct 30 19:42:38.698568 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:38.693389    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c89df4b-9274-4375-8ff8-47763bbf9182-istio-podinfo\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq\" (UID: \"3c89df4b-9274-4375-8ff8-47763bbf9182\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq"
Oct 30 19:42:38.698568 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:38.693457    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c89df4b-9274-4375-8ff8-47763bbf9182-workload-socket\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq\" (UID: \"3c89df4b-9274-4375-8ff8-47763bbf9182\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq"
Oct 30 19:42:38.698568 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:38.693490    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c89df4b-9274-4375-8ff8-47763bbf9182-istio-envoy\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq\" (UID: \"3c89df4b-9274-4375-8ff8-47763bbf9182\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq"
Oct 30 19:42:38.698568 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:38.693648    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk2sb\" (UniqueName: \"kubernetes.io/projected/3c89df4b-9274-4375-8ff8-47763bbf9182-kube-api-access-kk2sb\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq\" (UID: \"3c89df4b-9274-4375-8ff8-47763bbf9182\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq"
Oct 30 19:42:38.698568 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:38.694989    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c89df4b-9274-4375-8ff8-47763bbf9182-workload-certs\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq\" (UID: \"3c89df4b-9274-4375-8ff8-47763bbf9182\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq"
Oct 30 19:42:38.700948 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:38.700048    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c89df4b-9274-4375-8ff8-47763bbf9182-istio-data\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq\" (UID: \"3c89df4b-9274-4375-8ff8-47763bbf9182\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq"
Oct 30 19:42:38.700948 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:38.700534    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c89df4b-9274-4375-8ff8-47763bbf9182-credential-socket\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq\" (UID: \"3c89df4b-9274-4375-8ff8-47763bbf9182\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq"
Oct 30 19:42:38.701459 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:38.701438    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c89df4b-9274-4375-8ff8-47763bbf9182-workload-socket\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq\" (UID: \"3c89df4b-9274-4375-8ff8-47763bbf9182\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq"
Oct 30 19:42:38.702931 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:38.702907    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c89df4b-9274-4375-8ff8-47763bbf9182-istiod-ca-cert\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq\" (UID: \"3c89df4b-9274-4375-8ff8-47763bbf9182\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq"
Oct 30 19:42:38.711413 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:38.709884    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c89df4b-9274-4375-8ff8-47763bbf9182-istio-token\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq\" (UID: \"3c89df4b-9274-4375-8ff8-47763bbf9182\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq"
Oct 30 19:42:38.711790 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:38.711722    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c89df4b-9274-4375-8ff8-47763bbf9182-istio-envoy\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq\" (UID: \"3c89df4b-9274-4375-8ff8-47763bbf9182\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq"
Oct 30 19:42:38.712326 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:38.712294    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c89df4b-9274-4375-8ff8-47763bbf9182-istio-podinfo\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq\" (UID: \"3c89df4b-9274-4375-8ff8-47763bbf9182\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq"
Oct 30 19:42:38.713637 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:38.713615    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk2sb\" (UniqueName: \"kubernetes.io/projected/3c89df4b-9274-4375-8ff8-47763bbf9182-kube-api-access-kk2sb\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq\" (UID: \"3c89df4b-9274-4375-8ff8-47763bbf9182\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq"
Oct 30 19:42:38.767646 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:38.767614    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq"
Oct 30 19:42:39.040335 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:39.040299    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6266b58-dd5hf"]
Oct 30 19:42:39.045013 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:39.044972    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-194e54e4-9dd4p"]
Oct 30 19:42:39.049719 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:39.049694    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6266b58-dd5hf"]
Oct 30 19:42:39.053448 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:39.053424    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-194e54e4-9dd4p"]
Oct 30 19:42:39.074281 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:39.074257    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq"]
Oct 30 19:42:39.075592 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:42:39.075566    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c89df4b_9274_4375_8ff8_47763bbf9182.slice/crio-e5659a68adc7c15c5fa236a2346c671d132a82be1036ef7252e34afb0abe7feb WatchSource:0}: Error finding container e5659a68adc7c15c5fa236a2346c671d132a82be1036ef7252e34afb0abe7feb: Status 404 returned error can't find the container with id e5659a68adc7c15c5fa236a2346c671d132a82be1036ef7252e34afb0abe7feb
Oct 30 19:42:39.078549 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:39.078513    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:42:39.078667 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:39.078591    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:42:39.078667 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:39.078637    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:42:39.801082 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:39.801045    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq" event={"ID":"3c89df4b-9274-4375-8ff8-47763bbf9182","Type":"ContainerStarted","Data":"713483e9da389aa486b8e73e69d2718461ac01515c9c1c6aa61da6873afbc446"}
Oct 30 19:42:39.801752 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:39.801720    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq" event={"ID":"3c89df4b-9274-4375-8ff8-47763bbf9182","Type":"ContainerStarted","Data":"e5659a68adc7c15c5fa236a2346c671d132a82be1036ef7252e34afb0abe7feb"}
Oct 30 19:42:39.826623 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:39.826558    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq" podStartSLOduration=1.8265399150000001 podStartE2EDuration="1.826539915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:39.823596914 +0000 UTC m=+1579.334656879" watchObservedRunningTime="2026-10-30 19:42:39.826539915 +0000 UTC m=+1579.337599881"
Oct 30 19:42:40.646637 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:40.646602    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08851e5a-2c32-446b-9ffa-7508ac46cf2c" path="/var/lib/kubelet/pods/08851e5a-2c32-446b-9ffa-7508ac46cf2c/volumes"
Oct 30 19:42:40.647153 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:40.647128    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87ce476f-991e-4437-a15e-f7410d2392a5" path="/var/lib/kubelet/pods/87ce476f-991e-4437-a15e-f7410d2392a5/volumes"
Oct 30 19:42:40.768067 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:40.767996    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq"
Oct 30 19:42:40.775173 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:40.774829    5043 patch_prober.go:29] interesting pod/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Oct 30 19:42:40.775173 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:40.774883    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq" podUID="3c89df4b-9274-4375-8ff8-47763bbf9182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Oct 30 19:42:41.776763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:41.769062    5043 patch_prober.go:29] interesting pod/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Oct 30 19:42:41.776763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:41.769131    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq" podUID="3c89df4b-9274-4375-8ff8-47763bbf9182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Oct 30 19:42:42.774762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:42.773836    5043 patch_prober.go:29] interesting pod/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Oct 30 19:42:42.774762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:42.773901    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq" podUID="3c89df4b-9274-4375-8ff8-47763bbf9182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Oct 30 19:42:43.768246 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:43.768204    5043 patch_prober.go:29] interesting pod/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Oct 30 19:42:43.770330 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:43.770297    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq" podUID="3c89df4b-9274-4375-8ff8-47763bbf9182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Oct 30 19:42:44.770508 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:44.770442    5043 patch_prober.go:29] interesting pod/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Oct 30 19:42:44.771165 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:44.771118    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq" podUID="3c89df4b-9274-4375-8ff8-47763bbf9182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Oct 30 19:42:44.816776 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:44.813787    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b623ca-j9hhm"]
Oct 30 19:42:44.840199 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:44.840159    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-b623ca-j9hhm"]
Oct 30 19:42:45.781848 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:45.781808    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq"
Oct 30 19:42:45.783188 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:45.783166    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq"
Oct 30 19:42:45.784078 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:45.784047    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq"
Oct 30 19:42:46.646278 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:46.646244    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c8baf37-eb48-49c1-b373-c358ae9ae548" path="/var/lib/kubelet/pods/8c8baf37-eb48-49c1-b373-c358ae9ae548/volumes"
Oct 30 19:42:54.031180 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:54.031113    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb"]
Oct 30 19:42:54.033013 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:54.031378    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb" podUID="f43b74d6-8c5b-430d-bffb-a30beb6996ec" containerName="istio-proxy" containerID="cri-o://2ba4d9175cbdb92892dec3a9fe3684e7413884e3fd7380ded51b886fef7fe7ec" gracePeriod=30
Oct 30 19:42:54.979386 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:42:54.979341    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:42:59.356046 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:59.356018    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb"
Oct 30 19:42:59.535053 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:59.535018    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f43b74d6-8c5b-430d-bffb-a30beb6996ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f43b74d6-8c5b-430d-bffb-a30beb6996ec-istiod-ca-cert\") pod \"f43b74d6-8c5b-430d-bffb-a30beb6996ec\" (UID: \"f43b74d6-8c5b-430d-bffb-a30beb6996ec\") "
Oct 30 19:42:59.535252 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:59.535144    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f43b74d6-8c5b-430d-bffb-a30beb6996ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f43b74d6-8c5b-430d-bffb-a30beb6996ec-istio-envoy\") pod \"f43b74d6-8c5b-430d-bffb-a30beb6996ec\" (UID: \"f43b74d6-8c5b-430d-bffb-a30beb6996ec\") "
Oct 30 19:42:59.535252 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:59.535174    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f43b74d6-8c5b-430d-bffb-a30beb6996ec-istio-token\" (UniqueName: \"kubernetes.io/projected/f43b74d6-8c5b-430d-bffb-a30beb6996ec-istio-token\") pod \"f43b74d6-8c5b-430d-bffb-a30beb6996ec\" (UID: \"f43b74d6-8c5b-430d-bffb-a30beb6996ec\") "
Oct 30 19:42:59.535252 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:59.535206    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f43b74d6-8c5b-430d-bffb-a30beb6996ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43b74d6-8c5b-430d-bffb-a30beb6996ec-credential-socket\") pod \"f43b74d6-8c5b-430d-bffb-a30beb6996ec\" (UID: \"f43b74d6-8c5b-430d-bffb-a30beb6996ec\") "
Oct 30 19:42:59.535252 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:59.535245    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f43b74d6-8c5b-430d-bffb-a30beb6996ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f43b74d6-8c5b-430d-bffb-a30beb6996ec-istio-podinfo\") pod \"f43b74d6-8c5b-430d-bffb-a30beb6996ec\" (UID: \"f43b74d6-8c5b-430d-bffb-a30beb6996ec\") "
Oct 30 19:42:59.535497 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:59.535293    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f43b74d6-8c5b-430d-bffb-a30beb6996ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f43b74d6-8c5b-430d-bffb-a30beb6996ec-istio-data\") pod \"f43b74d6-8c5b-430d-bffb-a30beb6996ec\" (UID: \"f43b74d6-8c5b-430d-bffb-a30beb6996ec\") "
Oct 30 19:42:59.535497 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:59.535337    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f43b74d6-8c5b-430d-bffb-a30beb6996ec-kube-api-access-pvb89\" (UniqueName: \"kubernetes.io/projected/f43b74d6-8c5b-430d-bffb-a30beb6996ec-kube-api-access-pvb89\") pod \"f43b74d6-8c5b-430d-bffb-a30beb6996ec\" (UID: \"f43b74d6-8c5b-430d-bffb-a30beb6996ec\") "
Oct 30 19:42:59.535497 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:59.535371    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f43b74d6-8c5b-430d-bffb-a30beb6996ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f43b74d6-8c5b-430d-bffb-a30beb6996ec-workload-certs\") pod \"f43b74d6-8c5b-430d-bffb-a30beb6996ec\" (UID: \"f43b74d6-8c5b-430d-bffb-a30beb6996ec\") "
Oct 30 19:42:59.535497 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:59.535417    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f43b74d6-8c5b-430d-bffb-a30beb6996ec-istiod-ca-cert" pod "f43b74d6-8c5b-430d-bffb-a30beb6996ec" (UID: "f43b74d6-8c5b-430d-bffb-a30beb6996ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:42:59.535497 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:59.535482    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f43b74d6-8c5b-430d-bffb-a30beb6996ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43b74d6-8c5b-430d-bffb-a30beb6996ec-workload-socket\") pod \"f43b74d6-8c5b-430d-bffb-a30beb6996ec\" (UID: \"f43b74d6-8c5b-430d-bffb-a30beb6996ec\") "
Oct 30 19:42:59.536356 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:59.536028    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43b74d6-8c5b-430d-bffb-a30beb6996ec-credential-socket" pod "f43b74d6-8c5b-430d-bffb-a30beb6996ec" (UID: "f43b74d6-8c5b-430d-bffb-a30beb6996ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:42:59.536356 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:59.536216    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43b74d6-8c5b-430d-bffb-a30beb6996ec-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:59.536356 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:59.536224    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43b74d6-8c5b-430d-bffb-a30beb6996ec-workload-certs" pod "f43b74d6-8c5b-430d-bffb-a30beb6996ec" (UID: "f43b74d6-8c5b-430d-bffb-a30beb6996ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:42:59.536356 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:59.536236    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f43b74d6-8c5b-430d-bffb-a30beb6996ec-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:59.536356 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:59.536302    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43b74d6-8c5b-430d-bffb-a30beb6996ec-istio-data" pod "f43b74d6-8c5b-430d-bffb-a30beb6996ec" (UID: "f43b74d6-8c5b-430d-bffb-a30beb6996ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:42:59.537029 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:59.536471    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43b74d6-8c5b-430d-bffb-a30beb6996ec-workload-socket" pod "f43b74d6-8c5b-430d-bffb-a30beb6996ec" (UID: "f43b74d6-8c5b-430d-bffb-a30beb6996ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:42:59.538612 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:59.538586    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f43b74d6-8c5b-430d-bffb-a30beb6996ec-kube-api-access-pvb89" pod "f43b74d6-8c5b-430d-bffb-a30beb6996ec" (UID: "f43b74d6-8c5b-430d-bffb-a30beb6996ec"). InnerVolumeSpecName "kube-api-access-pvb89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:59.538717 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:59.538624    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f43b74d6-8c5b-430d-bffb-a30beb6996ec-istio-token" pod "f43b74d6-8c5b-430d-bffb-a30beb6996ec" (UID: "f43b74d6-8c5b-430d-bffb-a30beb6996ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:59.539599 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:59.539522    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43b74d6-8c5b-430d-bffb-a30beb6996ec-istio-envoy" pod "f43b74d6-8c5b-430d-bffb-a30beb6996ec" (UID: "f43b74d6-8c5b-430d-bffb-a30beb6996ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:42:59.539919 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:59.539892    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f43b74d6-8c5b-430d-bffb-a30beb6996ec-istio-podinfo" pod "f43b74d6-8c5b-430d-bffb-a30beb6996ec" (UID: "f43b74d6-8c5b-430d-bffb-a30beb6996ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:42:59.637693 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:59.637601    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f43b74d6-8c5b-430d-bffb-a30beb6996ec-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:59.637693 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:59.637630    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f43b74d6-8c5b-430d-bffb-a30beb6996ec-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:59.637693 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:59.637645    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f43b74d6-8c5b-430d-bffb-a30beb6996ec-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:59.637693 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:59.637657    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f43b74d6-8c5b-430d-bffb-a30beb6996ec-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:59.637693 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:59.637669    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvb89\" (UniqueName: \"kubernetes.io/projected/f43b74d6-8c5b-430d-bffb-a30beb6996ec-kube-api-access-pvb89\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:59.637693 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:59.637682    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f43b74d6-8c5b-430d-bffb-a30beb6996ec-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:59.637693 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:42:59.637694    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f43b74d6-8c5b-430d-bffb-a30beb6996ec-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:00.099156 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:00.099114    5043 generic.go:356] "Generic (PLEG): container finished" podID="f43b74d6-8c5b-430d-bffb-a30beb6996ec" containerID="2ba4d9175cbdb92892dec3a9fe3684e7413884e3fd7380ded51b886fef7fe7ec" exitCode=0
Oct 30 19:43:00.099372 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:00.099190    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb"
Oct 30 19:43:00.099372 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:00.099195    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb" event={"ID":"f43b74d6-8c5b-430d-bffb-a30beb6996ec","Type":"ContainerDied","Data":"2ba4d9175cbdb92892dec3a9fe3684e7413884e3fd7380ded51b886fef7fe7ec"}
Oct 30 19:43:00.099372 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:00.099301    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb" event={"ID":"f43b74d6-8c5b-430d-bffb-a30beb6996ec","Type":"ContainerDied","Data":"cab3a751b89d9ffd77985f0511bb7f300c6b018e3096a3e4c8820ba171a4a268"}
Oct 30 19:43:00.099372 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:00.099331    5043 scope.go:122] "RemoveContainer" containerID="2ba4d9175cbdb92892dec3a9fe3684e7413884e3fd7380ded51b886fef7fe7ec"
Oct 30 19:43:00.112158 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:00.112138    5043 scope.go:122] "RemoveContainer" containerID="2ba4d9175cbdb92892dec3a9fe3684e7413884e3fd7380ded51b886fef7fe7ec"
Oct 30 19:43:00.112524 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:43:00.112502    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ba4d9175cbdb92892dec3a9fe3684e7413884e3fd7380ded51b886fef7fe7ec\": container with ID starting with 2ba4d9175cbdb92892dec3a9fe3684e7413884e3fd7380ded51b886fef7fe7ec not found: ID does not exist" containerID="2ba4d9175cbdb92892dec3a9fe3684e7413884e3fd7380ded51b886fef7fe7ec"
Oct 30 19:43:00.112595 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:00.112534    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ba4d9175cbdb92892dec3a9fe3684e7413884e3fd7380ded51b886fef7fe7ec"} err="failed to get container status \"2ba4d9175cbdb92892dec3a9fe3684e7413884e3fd7380ded51b886fef7fe7ec\": rpc error: code = NotFound desc = could not find container \"2ba4d9175cbdb92892dec3a9fe3684e7413884e3fd7380ded51b886fef7fe7ec\": container with ID starting with 2ba4d9175cbdb92892dec3a9fe3684e7413884e3fd7380ded51b886fef7fe7ec not found: ID does not exist"
Oct 30 19:43:00.125577 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:00.125552    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb"]
Oct 30 19:43:00.129271 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:00.129246    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-rbvpb"]
Oct 30 19:43:00.647547 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:00.647511    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f43b74d6-8c5b-430d-bffb-a30beb6996ec" path="/var/lib/kubelet/pods/f43b74d6-8c5b-430d-bffb-a30beb6996ec/volumes"
Oct 30 19:43:02.362615 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:02.362584    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab530c70-xwg6v"]
Oct 30 19:43:02.363700 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:02.363677    5043 cpu_manager.go:414] "containerMap: removing container" podUID="f43b74d6-8c5b-430d-bffb-a30beb6996ec" containerName="istio-proxy"
Oct 30 19:43:02.363770 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:02.363703    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f43b74d6-8c5b-430d-bffb-a30beb6996ec" containerName="istio-proxy"
Oct 30 19:43:02.363891 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:02.363878    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="f43b74d6-8c5b-430d-bffb-a30beb6996ec" containerName="istio-proxy"
Oct 30 19:43:02.369765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:02.369710    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab530c70-xwg6v"
Oct 30 19:43:02.376578 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:02.375618    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:43:02.376578 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:02.375678    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:43:02.376578 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:02.375828    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:43:02.376578 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:02.375868    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:43:02.377508 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:02.377305    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab530c70-xwg6v"]
Oct 30 19:43:02.479527 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:02.479498    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24qdc\" (UniqueName: \"kubernetes.io/projected/e6372f52-7e2d-4118-82f7-8b9a20e5c0fc-kube-api-access-24qdc\") pod \"maas-api-revoke-sub-ab530c70-xwg6v\" (UID: \"e6372f52-7e2d-4118-82f7-8b9a20e5c0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab530c70-xwg6v"
Oct 30 19:43:02.479527 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:02.479530    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6372f52-7e2d-4118-82f7-8b9a20e5c0fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab530c70-xwg6v\" (UID: \"e6372f52-7e2d-4118-82f7-8b9a20e5c0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab530c70-xwg6v"
Oct 30 19:43:02.480114 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:02.479573    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6372f52-7e2d-4118-82f7-8b9a20e5c0fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab530c70-xwg6v\" (UID: \"e6372f52-7e2d-4118-82f7-8b9a20e5c0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab530c70-xwg6v"
Oct 30 19:43:02.581751 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:02.581690    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24qdc\" (UniqueName: \"kubernetes.io/projected/e6372f52-7e2d-4118-82f7-8b9a20e5c0fc-kube-api-access-24qdc\") pod \"maas-api-revoke-sub-ab530c70-xwg6v\" (UID: \"e6372f52-7e2d-4118-82f7-8b9a20e5c0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab530c70-xwg6v"
Oct 30 19:43:02.581751 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:02.581755    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6372f52-7e2d-4118-82f7-8b9a20e5c0fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab530c70-xwg6v\" (UID: \"e6372f52-7e2d-4118-82f7-8b9a20e5c0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab530c70-xwg6v"
Oct 30 19:43:02.582042 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:02.581823    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6372f52-7e2d-4118-82f7-8b9a20e5c0fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab530c70-xwg6v\" (UID: \"e6372f52-7e2d-4118-82f7-8b9a20e5c0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab530c70-xwg6v"
Oct 30 19:43:02.582671 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:02.582642    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6372f52-7e2d-4118-82f7-8b9a20e5c0fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab530c70-xwg6v\" (UID: \"e6372f52-7e2d-4118-82f7-8b9a20e5c0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab530c70-xwg6v"
Oct 30 19:43:02.584281 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:02.584257    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6372f52-7e2d-4118-82f7-8b9a20e5c0fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab530c70-xwg6v\" (UID: \"e6372f52-7e2d-4118-82f7-8b9a20e5c0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab530c70-xwg6v"
Oct 30 19:43:02.590972 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:02.590951    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24qdc\" (UniqueName: \"kubernetes.io/projected/e6372f52-7e2d-4118-82f7-8b9a20e5c0fc-kube-api-access-24qdc\") pod \"maas-api-revoke-sub-ab530c70-xwg6v\" (UID: \"e6372f52-7e2d-4118-82f7-8b9a20e5c0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab530c70-xwg6v"
Oct 30 19:43:02.707107 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:02.707018    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab530c70-xwg6v"
Oct 30 19:43:02.997443 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:02.997406    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab530c70-xwg6v"]
Oct 30 19:43:02.998106 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:43:02.998078    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6372f52_7e2d_4118_82f7_8b9a20e5c0fc.slice/crio-d3481c964e60eb2592b6e997aaee9bcc6bc4cc5f7d00e5406c262c2b5fdc7ff3 WatchSource:0}: Error finding container d3481c964e60eb2592b6e997aaee9bcc6bc4cc5f7d00e5406c262c2b5fdc7ff3: Status 404 returned error can't find the container with id d3481c964e60eb2592b6e997aaee9bcc6bc4cc5f7d00e5406c262c2b5fdc7ff3
Oct 30 19:43:03.029650 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:03.029620    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56581f7b-wzmj8"]
Oct 30 19:43:03.033377 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:03.033353    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56581f7b-wzmj8"]
Oct 30 19:43:03.141326 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:03.141284    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab530c70-xwg6v" event={"ID":"e6372f52-7e2d-4118-82f7-8b9a20e5c0fc","Type":"ContainerStarted","Data":"d3481c964e60eb2592b6e997aaee9bcc6bc4cc5f7d00e5406c262c2b5fdc7ff3"}
Oct 30 19:43:04.184760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:04.182003    5043 generic.go:356] "Generic (PLEG): container finished" podID="e6372f52-7e2d-4118-82f7-8b9a20e5c0fc" containerID="e42e23fb73a05f0c625b3dd00d9ffeca4fa0d9d52c2bb88d45a6b15eb0eb0206" exitCode=0
Oct 30 19:43:04.184760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:04.182403    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab530c70-xwg6v" event={"ID":"e6372f52-7e2d-4118-82f7-8b9a20e5c0fc","Type":"ContainerDied","Data":"e42e23fb73a05f0c625b3dd00d9ffeca4fa0d9d52c2bb88d45a6b15eb0eb0206"}
Oct 30 19:43:04.670950 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:04.670913    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162" path="/var/lib/kubelet/pods/6a87ff2b-ff7d-4e7c-b2ca-f1c78cc4f162/volumes"
Oct 30 19:43:05.402822 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:05.402800    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab530c70-xwg6v"
Oct 30 19:43:05.435285 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:05.434203    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6372f52-7e2d-4118-82f7-8b9a20e5c0fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6372f52-7e2d-4118-82f7-8b9a20e5c0fc-maas-api-xks-ca\") pod \"e6372f52-7e2d-4118-82f7-8b9a20e5c0fc\" (UID: \"e6372f52-7e2d-4118-82f7-8b9a20e5c0fc\") "
Oct 30 19:43:05.435285 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:05.434357    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6372f52-7e2d-4118-82f7-8b9a20e5c0fc-kube-api-access-24qdc\" (UniqueName: \"kubernetes.io/projected/e6372f52-7e2d-4118-82f7-8b9a20e5c0fc-kube-api-access-24qdc\") pod \"e6372f52-7e2d-4118-82f7-8b9a20e5c0fc\" (UID: \"e6372f52-7e2d-4118-82f7-8b9a20e5c0fc\") "
Oct 30 19:43:05.435285 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:05.434512    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6372f52-7e2d-4118-82f7-8b9a20e5c0fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6372f52-7e2d-4118-82f7-8b9a20e5c0fc-maas-api-openshift-ca\") pod \"e6372f52-7e2d-4118-82f7-8b9a20e5c0fc\" (UID: \"e6372f52-7e2d-4118-82f7-8b9a20e5c0fc\") "
Oct 30 19:43:05.435285 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:05.434956    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6372f52-7e2d-4118-82f7-8b9a20e5c0fc-maas-api-openshift-ca" pod "e6372f52-7e2d-4118-82f7-8b9a20e5c0fc" (UID: "e6372f52-7e2d-4118-82f7-8b9a20e5c0fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:05.435285 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:05.435249    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6372f52-7e2d-4118-82f7-8b9a20e5c0fc-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:05.439179 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:05.439139    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6372f52-7e2d-4118-82f7-8b9a20e5c0fc-maas-api-xks-ca" pod "e6372f52-7e2d-4118-82f7-8b9a20e5c0fc" (UID: "e6372f52-7e2d-4118-82f7-8b9a20e5c0fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:05.439477 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:05.439449    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6372f52-7e2d-4118-82f7-8b9a20e5c0fc-kube-api-access-24qdc" pod "e6372f52-7e2d-4118-82f7-8b9a20e5c0fc" (UID: "e6372f52-7e2d-4118-82f7-8b9a20e5c0fc"). InnerVolumeSpecName "kube-api-access-24qdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:05.536430 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:05.536397    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6372f52-7e2d-4118-82f7-8b9a20e5c0fc-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:05.536430 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:05.536430    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24qdc\" (UniqueName: \"kubernetes.io/projected/e6372f52-7e2d-4118-82f7-8b9a20e5c0fc-kube-api-access-24qdc\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:06.205261 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:06.205221    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab530c70-xwg6v" event={"ID":"e6372f52-7e2d-4118-82f7-8b9a20e5c0fc","Type":"ContainerDied","Data":"d3481c964e60eb2592b6e997aaee9bcc6bc4cc5f7d00e5406c262c2b5fdc7ff3"}
Oct 30 19:43:06.205261 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:06.205248    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab530c70-xwg6v"
Oct 30 19:43:06.205541 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:06.205280    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3481c964e60eb2592b6e997aaee9bcc6bc4cc5f7d00e5406c262c2b5fdc7ff3"
Oct 30 19:43:12.967520 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:12.967488    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-579ddb/test-model-579ddb-kserve-69c7664c5c-nllwv"]
Oct 30 19:43:13.479648 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:13.479617    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f06bdefe-8blkv"]
Oct 30 19:43:13.480409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:13.480391    5043 cpu_manager.go:414] "containerMap: removing container" podUID="e6372f52-7e2d-4118-82f7-8b9a20e5c0fc" containerName="revoke-keys"
Oct 30 19:43:13.480504 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:13.480411    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6372f52-7e2d-4118-82f7-8b9a20e5c0fc" containerName="revoke-keys"
Oct 30 19:43:13.480595 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:13.480583    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6372f52-7e2d-4118-82f7-8b9a20e5c0fc" containerName="revoke-keys"
Oct 30 19:43:13.484578 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:13.484554    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06bdefe-8blkv"
Oct 30 19:43:13.491910 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:13.491886    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:43:13.493056 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:13.491897    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:43:13.498584 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:13.498560    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f06bdefe-8blkv"]
Oct 30 19:43:13.534934 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:13.534897    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64f45468-8b8a-4191-a217-b7b6993d94ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f06bdefe-8blkv\" (UID: \"64f45468-8b8a-4191-a217-b7b6993d94ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06bdefe-8blkv"
Oct 30 19:43:13.535139 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:13.534973    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64f45468-8b8a-4191-a217-b7b6993d94ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f06bdefe-8blkv\" (UID: \"64f45468-8b8a-4191-a217-b7b6993d94ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06bdefe-8blkv"
Oct 30 19:43:13.535139 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:13.535080    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8htj\" (UniqueName: \"kubernetes.io/projected/64f45468-8b8a-4191-a217-b7b6993d94ed-kube-api-access-h8htj\") pod \"maas-api-revoke-sub-f06bdefe-8blkv\" (UID: \"64f45468-8b8a-4191-a217-b7b6993d94ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06bdefe-8blkv"
Oct 30 19:43:13.538955 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:13.538932    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:43:13.539156 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:13.539046    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:43:13.636870 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:13.636831    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64f45468-8b8a-4191-a217-b7b6993d94ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f06bdefe-8blkv\" (UID: \"64f45468-8b8a-4191-a217-b7b6993d94ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06bdefe-8blkv"
Oct 30 19:43:13.637069 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:13.636933    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64f45468-8b8a-4191-a217-b7b6993d94ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f06bdefe-8blkv\" (UID: \"64f45468-8b8a-4191-a217-b7b6993d94ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06bdefe-8blkv"
Oct 30 19:43:13.637069 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:13.636972    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8htj\" (UniqueName: \"kubernetes.io/projected/64f45468-8b8a-4191-a217-b7b6993d94ed-kube-api-access-h8htj\") pod \"maas-api-revoke-sub-f06bdefe-8blkv\" (UID: \"64f45468-8b8a-4191-a217-b7b6993d94ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06bdefe-8blkv"
Oct 30 19:43:13.637536 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:13.637508    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64f45468-8b8a-4191-a217-b7b6993d94ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f06bdefe-8blkv\" (UID: \"64f45468-8b8a-4191-a217-b7b6993d94ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06bdefe-8blkv"
Oct 30 19:43:13.639624 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:13.639603    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64f45468-8b8a-4191-a217-b7b6993d94ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f06bdefe-8blkv\" (UID: \"64f45468-8b8a-4191-a217-b7b6993d94ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06bdefe-8blkv"
Oct 30 19:43:13.646505 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:13.646485    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8htj\" (UniqueName: \"kubernetes.io/projected/64f45468-8b8a-4191-a217-b7b6993d94ed-kube-api-access-h8htj\") pod \"maas-api-revoke-sub-f06bdefe-8blkv\" (UID: \"64f45468-8b8a-4191-a217-b7b6993d94ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06bdefe-8blkv"
Oct 30 19:43:13.805055 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:13.804964    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06bdefe-8blkv"
Oct 30 19:43:14.093419 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:14.093382    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f06bdefe-8blkv"]
Oct 30 19:43:14.094136 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:43:14.094107    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64f45468_8b8a_4191_a217_b7b6993d94ed.slice/crio-3d8188891a214de3f9def236eebe70a89f6c72571a47b5bdfeec2ecbf3090055 WatchSource:0}: Error finding container 3d8188891a214de3f9def236eebe70a89f6c72571a47b5bdfeec2ecbf3090055: Status 404 returned error can't find the container with id 3d8188891a214de3f9def236eebe70a89f6c72571a47b5bdfeec2ecbf3090055
Oct 30 19:43:14.293326 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:14.293289    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06bdefe-8blkv" event={"ID":"64f45468-8b8a-4191-a217-b7b6993d94ed","Type":"ContainerStarted","Data":"3d8188891a214de3f9def236eebe70a89f6c72571a47b5bdfeec2ecbf3090055"}
Oct 30 19:43:15.304887 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:15.304794    5043 generic.go:356] "Generic (PLEG): container finished" podID="64f45468-8b8a-4191-a217-b7b6993d94ed" containerID="8849943ec72f7ac027b43c6d7883e866bf9cbb2d6cacad33d734aa168cf7624d" exitCode=0
Oct 30 19:43:15.305370 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:15.304876    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06bdefe-8blkv" event={"ID":"64f45468-8b8a-4191-a217-b7b6993d94ed","Type":"ContainerDied","Data":"8849943ec72f7ac027b43c6d7883e866bf9cbb2d6cacad33d734aa168cf7624d"}
Oct 30 19:43:16.509843 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:16.509820    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06bdefe-8blkv"
Oct 30 19:43:16.589065 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:16.588980    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64f45468-8b8a-4191-a217-b7b6993d94ed-kube-api-access-h8htj\" (UniqueName: \"kubernetes.io/projected/64f45468-8b8a-4191-a217-b7b6993d94ed-kube-api-access-h8htj\") pod \"64f45468-8b8a-4191-a217-b7b6993d94ed\" (UID: \"64f45468-8b8a-4191-a217-b7b6993d94ed\") "
Oct 30 19:43:16.589248 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:16.589125    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64f45468-8b8a-4191-a217-b7b6993d94ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64f45468-8b8a-4191-a217-b7b6993d94ed-maas-api-xks-ca\") pod \"64f45468-8b8a-4191-a217-b7b6993d94ed\" (UID: \"64f45468-8b8a-4191-a217-b7b6993d94ed\") "
Oct 30 19:43:16.589480 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:16.589457    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64f45468-8b8a-4191-a217-b7b6993d94ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64f45468-8b8a-4191-a217-b7b6993d94ed-maas-api-openshift-ca\") pod \"64f45468-8b8a-4191-a217-b7b6993d94ed\" (UID: \"64f45468-8b8a-4191-a217-b7b6993d94ed\") "
Oct 30 19:43:16.590419 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:16.590395    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64f45468-8b8a-4191-a217-b7b6993d94ed-maas-api-openshift-ca" pod "64f45468-8b8a-4191-a217-b7b6993d94ed" (UID: "64f45468-8b8a-4191-a217-b7b6993d94ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:16.591834 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:16.591807    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64f45468-8b8a-4191-a217-b7b6993d94ed-maas-api-xks-ca" pod "64f45468-8b8a-4191-a217-b7b6993d94ed" (UID: "64f45468-8b8a-4191-a217-b7b6993d94ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:16.594002 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:16.593946    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64f45468-8b8a-4191-a217-b7b6993d94ed-kube-api-access-h8htj" pod "64f45468-8b8a-4191-a217-b7b6993d94ed" (UID: "64f45468-8b8a-4191-a217-b7b6993d94ed"). InnerVolumeSpecName "kube-api-access-h8htj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:16.691859 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:16.691755    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8htj\" (UniqueName: \"kubernetes.io/projected/64f45468-8b8a-4191-a217-b7b6993d94ed-kube-api-access-h8htj\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:16.691859 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:16.691790    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64f45468-8b8a-4191-a217-b7b6993d94ed-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:16.691859 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:16.691804    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64f45468-8b8a-4191-a217-b7b6993d94ed-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:17.327171 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:17.327143    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06bdefe-8blkv"
Oct 30 19:43:17.327350 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:17.327187    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f06bdefe-8blkv" event={"ID":"64f45468-8b8a-4191-a217-b7b6993d94ed","Type":"ContainerDied","Data":"3d8188891a214de3f9def236eebe70a89f6c72571a47b5bdfeec2ecbf3090055"}
Oct 30 19:43:17.327350 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:17.327235    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d8188891a214de3f9def236eebe70a89f6c72571a47b5bdfeec2ecbf3090055"
Oct 30 19:43:24.218134 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:24.218097    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1e13df/test-model-1e13df-kserve-c8b994f5f-97xvv"]
Oct 30 19:43:24.351023 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:24.348142    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-579ddb-6qrtw"]
Oct 30 19:43:24.351023 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:24.349255    5043 cpu_manager.go:414] "containerMap: removing container" podUID="64f45468-8b8a-4191-a217-b7b6993d94ed" containerName="revoke-keys"
Oct 30 19:43:24.351023 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:24.349277    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64f45468-8b8a-4191-a217-b7b6993d94ed" containerName="revoke-keys"
Oct 30 19:43:24.351023 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:24.349451    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="64f45468-8b8a-4191-a217-b7b6993d94ed" containerName="revoke-keys"
Oct 30 19:43:24.354130 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:24.354100    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-579ddb-6qrtw"]
Oct 30 19:43:24.354353 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:24.354253    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-579ddb-6qrtw"
Oct 30 19:43:24.361165 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:24.361123    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:43:24.361335 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:24.361316    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:43:24.442515 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:24.433196    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:43:24.442515 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:24.433428    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:43:24.516706 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:24.516625    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/412bfb73-ee32-4740-83aa-3dacd1a2355a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-579ddb-6qrtw\" (UID: \"412bfb73-ee32-4740-83aa-3dacd1a2355a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-579ddb-6qrtw"
Oct 30 19:43:24.516884 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:24.516771    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8shxz\" (UniqueName: \"kubernetes.io/projected/412bfb73-ee32-4740-83aa-3dacd1a2355a-kube-api-access-8shxz\") pod \"maas-api-revoke-keys-e2e-inf-a-579ddb-6qrtw\" (UID: \"412bfb73-ee32-4740-83aa-3dacd1a2355a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-579ddb-6qrtw"
Oct 30 19:43:24.516936 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:24.516916    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/412bfb73-ee32-4740-83aa-3dacd1a2355a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-579ddb-6qrtw\" (UID: \"412bfb73-ee32-4740-83aa-3dacd1a2355a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-579ddb-6qrtw"
Oct 30 19:43:24.544453 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:24.544426    5043 scope.go:122] "RemoveContainer" containerID="da655624aeff9d702a50182e5e2dcbf4a7f888b465ae83e3b439fa5c0216db25"
Oct 30 19:43:24.555973 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:24.555951    5043 scope.go:122] "RemoveContainer" containerID="a7e04db9d36e78baa44b94fe553ddd8a5cc45890bd175369a55696bc33c0e5e4"
Oct 30 19:43:24.569821 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:24.569803    5043 scope.go:122] "RemoveContainer" containerID="8e344b169ec915ed117875b39c9e69b1b7b0ee875e23b94b62b13eb037410182"
Oct 30 19:43:24.581514 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:24.581492    5043 scope.go:122] "RemoveContainer" containerID="9a2fc964e42b10685d8154dc527ed1ca8d8411fc3aec890b938260d420107081"
Oct 30 19:43:24.619476 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:24.619407    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/412bfb73-ee32-4740-83aa-3dacd1a2355a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-579ddb-6qrtw\" (UID: \"412bfb73-ee32-4740-83aa-3dacd1a2355a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-579ddb-6qrtw"
Oct 30 19:43:24.619552 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:24.619505    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8shxz\" (UniqueName: \"kubernetes.io/projected/412bfb73-ee32-4740-83aa-3dacd1a2355a-kube-api-access-8shxz\") pod \"maas-api-revoke-keys-e2e-inf-a-579ddb-6qrtw\" (UID: \"412bfb73-ee32-4740-83aa-3dacd1a2355a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-579ddb-6qrtw"
Oct 30 19:43:24.619638 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:24.619619    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/412bfb73-ee32-4740-83aa-3dacd1a2355a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-579ddb-6qrtw\" (UID: \"412bfb73-ee32-4740-83aa-3dacd1a2355a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-579ddb-6qrtw"
Oct 30 19:43:24.620233 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:24.620209    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/412bfb73-ee32-4740-83aa-3dacd1a2355a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-579ddb-6qrtw\" (UID: \"412bfb73-ee32-4740-83aa-3dacd1a2355a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-579ddb-6qrtw"
Oct 30 19:43:24.622054 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:24.622032    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/412bfb73-ee32-4740-83aa-3dacd1a2355a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-579ddb-6qrtw\" (UID: \"412bfb73-ee32-4740-83aa-3dacd1a2355a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-579ddb-6qrtw"
Oct 30 19:43:24.628722 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:24.628701    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8shxz\" (UniqueName: \"kubernetes.io/projected/412bfb73-ee32-4740-83aa-3dacd1a2355a-kube-api-access-8shxz\") pod \"maas-api-revoke-keys-e2e-inf-a-579ddb-6qrtw\" (UID: \"412bfb73-ee32-4740-83aa-3dacd1a2355a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-579ddb-6qrtw"
Oct 30 19:43:24.695785 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:24.695724    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-579ddb-6qrtw"
Oct 30 19:43:24.982218 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:43:24.982168    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:43:25.065231 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:25.065194    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-579ddb-6qrtw"]
Oct 30 19:43:25.067198 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:43:25.067172    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod412bfb73_ee32_4740_83aa_3dacd1a2355a.slice/crio-ac6f57b6a7a9ce1ab3e14d2f972f2af3e40ebb93f125c0a6afd0e48d01d135c9 WatchSource:0}: Error finding container ac6f57b6a7a9ce1ab3e14d2f972f2af3e40ebb93f125c0a6afd0e48d01d135c9: Status 404 returned error can't find the container with id ac6f57b6a7a9ce1ab3e14d2f972f2af3e40ebb93f125c0a6afd0e48d01d135c9
Oct 30 19:43:25.445286 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:25.445203    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-579ddb-6qrtw" event={"ID":"412bfb73-ee32-4740-83aa-3dacd1a2355a","Type":"ContainerStarted","Data":"ac6f57b6a7a9ce1ab3e14d2f972f2af3e40ebb93f125c0a6afd0e48d01d135c9"}
Oct 30 19:43:26.455283 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:26.455252    5043 generic.go:356] "Generic (PLEG): container finished" podID="412bfb73-ee32-4740-83aa-3dacd1a2355a" containerID="08d60fe43e98fc3a5ed05f53713bf723a3c33a2ffd0133545201e9e72dcd2f2d" exitCode=0
Oct 30 19:43:26.455668 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:26.455328    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-579ddb-6qrtw" event={"ID":"412bfb73-ee32-4740-83aa-3dacd1a2355a","Type":"ContainerDied","Data":"08d60fe43e98fc3a5ed05f53713bf723a3c33a2ffd0133545201e9e72dcd2f2d"}
Oct 30 19:43:27.648928 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:27.648905    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-579ddb-6qrtw"
Oct 30 19:43:27.662253 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:27.662223    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/412bfb73-ee32-4740-83aa-3dacd1a2355a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/412bfb73-ee32-4740-83aa-3dacd1a2355a-maas-api-openshift-ca\") pod \"412bfb73-ee32-4740-83aa-3dacd1a2355a\" (UID: \"412bfb73-ee32-4740-83aa-3dacd1a2355a\") "
Oct 30 19:43:27.662403 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:27.662267    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/412bfb73-ee32-4740-83aa-3dacd1a2355a-kube-api-access-8shxz\" (UniqueName: \"kubernetes.io/projected/412bfb73-ee32-4740-83aa-3dacd1a2355a-kube-api-access-8shxz\") pod \"412bfb73-ee32-4740-83aa-3dacd1a2355a\" (UID: \"412bfb73-ee32-4740-83aa-3dacd1a2355a\") "
Oct 30 19:43:27.662403 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:27.662398    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/412bfb73-ee32-4740-83aa-3dacd1a2355a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/412bfb73-ee32-4740-83aa-3dacd1a2355a-maas-api-xks-ca\") pod \"412bfb73-ee32-4740-83aa-3dacd1a2355a\" (UID: \"412bfb73-ee32-4740-83aa-3dacd1a2355a\") "
Oct 30 19:43:27.662578 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:27.662550    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/412bfb73-ee32-4740-83aa-3dacd1a2355a-maas-api-openshift-ca" pod "412bfb73-ee32-4740-83aa-3dacd1a2355a" (UID: "412bfb73-ee32-4740-83aa-3dacd1a2355a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:27.663209 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:27.663183    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/412bfb73-ee32-4740-83aa-3dacd1a2355a-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:27.664631 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:27.664602    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/412bfb73-ee32-4740-83aa-3dacd1a2355a-kube-api-access-8shxz" pod "412bfb73-ee32-4740-83aa-3dacd1a2355a" (UID: "412bfb73-ee32-4740-83aa-3dacd1a2355a"). InnerVolumeSpecName "kube-api-access-8shxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:27.664724 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:27.664643    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/412bfb73-ee32-4740-83aa-3dacd1a2355a-maas-api-xks-ca" pod "412bfb73-ee32-4740-83aa-3dacd1a2355a" (UID: "412bfb73-ee32-4740-83aa-3dacd1a2355a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:27.782065 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:27.781975    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8shxz\" (UniqueName: \"kubernetes.io/projected/412bfb73-ee32-4740-83aa-3dacd1a2355a-kube-api-access-8shxz\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:27.782277 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:27.782259    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/412bfb73-ee32-4740-83aa-3dacd1a2355a-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:27.996304 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:27.996262    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-579ddb/test-model-579ddb-kserve-69c7664c5c-nllwv" podUID="cb491571-bc76-4a0b-b931-df9437d7a5bd" containerName="main" containerID="cri-o://61f24be4ad43e4a8745a334f4b6e54bcce10b91fcdd50b69ed0e308b31e25e89" gracePeriod=45
Oct 30 19:43:28.213302 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.213264    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41871c4d-4mpdm"]
Oct 30 19:43:28.214426 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.214399    5043 cpu_manager.go:414] "containerMap: removing container" podUID="412bfb73-ee32-4740-83aa-3dacd1a2355a" containerName="revoke-keys"
Oct 30 19:43:28.214426 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.214427    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="412bfb73-ee32-4740-83aa-3dacd1a2355a" containerName="revoke-keys"
Oct 30 19:43:28.214686 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.214668    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="412bfb73-ee32-4740-83aa-3dacd1a2355a" containerName="revoke-keys"
Oct 30 19:43:28.224578 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.221211    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41871c4d-4mpdm"
Oct 30 19:43:28.224578 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.224429    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41871c4d-4mpdm"]
Oct 30 19:43:28.282852 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.282824    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-579ddb/test-model-579ddb-kserve-69c7664c5c-nllwv"
Oct 30 19:43:28.293532 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.293495    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41871c4d-4mpdm\" (UID: \"aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41871c4d-4mpdm"
Oct 30 19:43:28.293691 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.293570    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41871c4d-4mpdm\" (UID: \"aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41871c4d-4mpdm"
Oct 30 19:43:28.293761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.293687    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjmx2\" (UniqueName: \"kubernetes.io/projected/aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5-kube-api-access-gjmx2\") pod \"maas-api-revoke-sub-41871c4d-4mpdm\" (UID: \"aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41871c4d-4mpdm"
Oct 30 19:43:28.394938 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.394851    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb491571-bc76-4a0b-b931-df9437d7a5bd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb491571-bc76-4a0b-b931-df9437d7a5bd-tmp-dir\") pod \"cb491571-bc76-4a0b-b931-df9437d7a5bd\" (UID: \"cb491571-bc76-4a0b-b931-df9437d7a5bd\") "
Oct 30 19:43:28.394938 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.394919    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb491571-bc76-4a0b-b931-df9437d7a5bd-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb491571-bc76-4a0b-b931-df9437d7a5bd-tls-certs\") pod \"cb491571-bc76-4a0b-b931-df9437d7a5bd\" (UID: \"cb491571-bc76-4a0b-b931-df9437d7a5bd\") "
Oct 30 19:43:28.395191 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.394976    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb491571-bc76-4a0b-b931-df9437d7a5bd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb491571-bc76-4a0b-b931-df9437d7a5bd-model-cache\") pod \"cb491571-bc76-4a0b-b931-df9437d7a5bd\" (UID: \"cb491571-bc76-4a0b-b931-df9437d7a5bd\") "
Oct 30 19:43:28.395191 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.395123    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb491571-bc76-4a0b-b931-df9437d7a5bd-home\" (UniqueName: \"kubernetes.io/empty-dir/cb491571-bc76-4a0b-b931-df9437d7a5bd-home\") pod \"cb491571-bc76-4a0b-b931-df9437d7a5bd\" (UID: \"cb491571-bc76-4a0b-b931-df9437d7a5bd\") "
Oct 30 19:43:28.395191 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.395131    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb491571-bc76-4a0b-b931-df9437d7a5bd-tmp-dir" pod "cb491571-bc76-4a0b-b931-df9437d7a5bd" (UID: "cb491571-bc76-4a0b-b931-df9437d7a5bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:28.395365 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.395212    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb491571-bc76-4a0b-b931-df9437d7a5bd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb491571-bc76-4a0b-b931-df9437d7a5bd-dshm\") pod \"cb491571-bc76-4a0b-b931-df9437d7a5bd\" (UID: \"cb491571-bc76-4a0b-b931-df9437d7a5bd\") "
Oct 30 19:43:28.395365 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.395260    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb491571-bc76-4a0b-b931-df9437d7a5bd-kube-api-access-trlzx\" (UniqueName: \"kubernetes.io/projected/cb491571-bc76-4a0b-b931-df9437d7a5bd-kube-api-access-trlzx\") pod \"cb491571-bc76-4a0b-b931-df9437d7a5bd\" (UID: \"cb491571-bc76-4a0b-b931-df9437d7a5bd\") "
Oct 30 19:43:28.395487 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.395400    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb491571-bc76-4a0b-b931-df9437d7a5bd-home" pod "cb491571-bc76-4a0b-b931-df9437d7a5bd" (UID: "cb491571-bc76-4a0b-b931-df9437d7a5bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:28.395607 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.395341    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb491571-bc76-4a0b-b931-df9437d7a5bd-model-cache" pod "cb491571-bc76-4a0b-b931-df9437d7a5bd" (UID: "cb491571-bc76-4a0b-b931-df9437d7a5bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:28.395792 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.395768    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41871c4d-4mpdm\" (UID: \"aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41871c4d-4mpdm"
Oct 30 19:43:28.395946 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.395853    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41871c4d-4mpdm\" (UID: \"aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41871c4d-4mpdm"
Oct 30 19:43:28.395946 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.395908    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjmx2\" (UniqueName: \"kubernetes.io/projected/aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5-kube-api-access-gjmx2\") pod \"maas-api-revoke-sub-41871c4d-4mpdm\" (UID: \"aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41871c4d-4mpdm"
Oct 30 19:43:28.396807 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.396075    5043 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb491571-bc76-4a0b-b931-df9437d7a5bd-tmp-dir\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.396807 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.396095    5043 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb491571-bc76-4a0b-b931-df9437d7a5bd-model-cache\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.396807 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.396110    5043 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb491571-bc76-4a0b-b931-df9437d7a5bd-home\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.396807 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.396801    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41871c4d-4mpdm\" (UID: \"aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41871c4d-4mpdm"
Oct 30 19:43:28.397881 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.397853    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb491571-bc76-4a0b-b931-df9437d7a5bd-tls-certs" pod "cb491571-bc76-4a0b-b931-df9437d7a5bd" (UID: "cb491571-bc76-4a0b-b931-df9437d7a5bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:28.398214 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.398191    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb491571-bc76-4a0b-b931-df9437d7a5bd-dshm" pod "cb491571-bc76-4a0b-b931-df9437d7a5bd" (UID: "cb491571-bc76-4a0b-b931-df9437d7a5bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:28.398285 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.398195    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb491571-bc76-4a0b-b931-df9437d7a5bd-kube-api-access-trlzx" pod "cb491571-bc76-4a0b-b931-df9437d7a5bd" (UID: "cb491571-bc76-4a0b-b931-df9437d7a5bd"). InnerVolumeSpecName "kube-api-access-trlzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:28.399179 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.399161    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41871c4d-4mpdm\" (UID: \"aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41871c4d-4mpdm"
Oct 30 19:43:28.405360 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.405340    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjmx2\" (UniqueName: \"kubernetes.io/projected/aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5-kube-api-access-gjmx2\") pod \"maas-api-revoke-sub-41871c4d-4mpdm\" (UID: \"aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41871c4d-4mpdm"
Oct 30 19:43:28.474705 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.474678    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-579ddb-6qrtw"
Oct 30 19:43:28.474907 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.474671    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-579ddb-6qrtw" event={"ID":"412bfb73-ee32-4740-83aa-3dacd1a2355a","Type":"ContainerDied","Data":"ac6f57b6a7a9ce1ab3e14d2f972f2af3e40ebb93f125c0a6afd0e48d01d135c9"}
Oct 30 19:43:28.474907 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.474875    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac6f57b6a7a9ce1ab3e14d2f972f2af3e40ebb93f125c0a6afd0e48d01d135c9"
Oct 30 19:43:28.476422 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.476395    5043 generic.go:356] "Generic (PLEG): container finished" podID="cb491571-bc76-4a0b-b931-df9437d7a5bd" containerID="61f24be4ad43e4a8745a334f4b6e54bcce10b91fcdd50b69ed0e308b31e25e89" exitCode=0
Oct 30 19:43:28.476571 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.476442    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-579ddb/test-model-579ddb-kserve-69c7664c5c-nllwv"
Oct 30 19:43:28.476571 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.476474    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-579ddb/test-model-579ddb-kserve-69c7664c5c-nllwv" event={"ID":"cb491571-bc76-4a0b-b931-df9437d7a5bd","Type":"ContainerDied","Data":"61f24be4ad43e4a8745a334f4b6e54bcce10b91fcdd50b69ed0e308b31e25e89"}
Oct 30 19:43:28.476571 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.476516    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-579ddb/test-model-579ddb-kserve-69c7664c5c-nllwv" event={"ID":"cb491571-bc76-4a0b-b931-df9437d7a5bd","Type":"ContainerDied","Data":"f99c04669e5751f8ca766e9f4a41be392c77fd4b3404e003725fe21717465c15"}
Oct 30 19:43:28.476571 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.476537    5043 scope.go:122] "RemoveContainer" containerID="61f24be4ad43e4a8745a334f4b6e54bcce10b91fcdd50b69ed0e308b31e25e89"
Oct 30 19:43:28.494141 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.494122    5043 scope.go:122] "RemoveContainer" containerID="61f24be4ad43e4a8745a334f4b6e54bcce10b91fcdd50b69ed0e308b31e25e89"
Oct 30 19:43:28.494411 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:43:28.494394    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61f24be4ad43e4a8745a334f4b6e54bcce10b91fcdd50b69ed0e308b31e25e89\": container with ID starting with 61f24be4ad43e4a8745a334f4b6e54bcce10b91fcdd50b69ed0e308b31e25e89 not found: ID does not exist" containerID="61f24be4ad43e4a8745a334f4b6e54bcce10b91fcdd50b69ed0e308b31e25e89"
Oct 30 19:43:28.494495 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.494419    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61f24be4ad43e4a8745a334f4b6e54bcce10b91fcdd50b69ed0e308b31e25e89"} err="failed to get container status \"61f24be4ad43e4a8745a334f4b6e54bcce10b91fcdd50b69ed0e308b31e25e89\": rpc error: code = NotFound desc = could not find container \"61f24be4ad43e4a8745a334f4b6e54bcce10b91fcdd50b69ed0e308b31e25e89\": container with ID starting with 61f24be4ad43e4a8745a334f4b6e54bcce10b91fcdd50b69ed0e308b31e25e89 not found: ID does not exist"
Oct 30 19:43:28.497593 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.497567    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trlzx\" (UniqueName: \"kubernetes.io/projected/cb491571-bc76-4a0b-b931-df9437d7a5bd-kube-api-access-trlzx\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.497679 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.497599    5043 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb491571-bc76-4a0b-b931-df9437d7a5bd-tls-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.497679 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.497615    5043 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb491571-bc76-4a0b-b931-df9437d7a5bd-dshm\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:28.507188 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.507147    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-579ddb/test-model-579ddb-kserve-69c7664c5c-nllwv"]
Oct 30 19:43:28.511161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.511144    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-579ddb/test-model-579ddb-kserve-69c7664c5c-nllwv"]
Oct 30 19:43:28.547870 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.547838    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41871c4d-4mpdm"
Oct 30 19:43:28.687350 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.686849    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb491571-bc76-4a0b-b931-df9437d7a5bd" path="/var/lib/kubelet/pods/cb491571-bc76-4a0b-b931-df9437d7a5bd/volumes"
Oct 30 19:43:28.865871 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:43:28.865827    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa0c5459_e25b_451b_ba4f_d2a6d9cd9ec5.slice/crio-a628c74692aad1f44928ad0131eee4ca1b1429223bb595ced24486bc1f176b14 WatchSource:0}: Error finding container a628c74692aad1f44928ad0131eee4ca1b1429223bb595ced24486bc1f176b14: Status 404 returned error can't find the container with id a628c74692aad1f44928ad0131eee4ca1b1429223bb595ced24486bc1f176b14
Oct 30 19:43:28.866026 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:28.865888    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41871c4d-4mpdm"]
Oct 30 19:43:29.489930 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:29.489900    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41871c4d-4mpdm" event={"ID":"aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5","Type":"ContainerStarted","Data":"70dbbc48a0dc64c23ddc584d5984228af148062593220caed5e0fdbca84e3fc0"}
Oct 30 19:43:29.490108 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:29.489960    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41871c4d-4mpdm" event={"ID":"aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5","Type":"ContainerStarted","Data":"a628c74692aad1f44928ad0131eee4ca1b1429223bb595ced24486bc1f176b14"}
Oct 30 19:43:29.513099 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:29.513047    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41871c4d-4mpdm" podStartSLOduration=1.5130342030000001 podStartE2EDuration="1.513034203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:29.509701485 +0000 UTC m=+1629.020761450" watchObservedRunningTime="2026-10-30 19:43:29.513034203 +0000 UTC m=+1629.024094445"
Oct 30 19:43:30.521030 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:30.520993    5043 generic.go:356] "Generic (PLEG): container finished" podID="aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5" containerID="70dbbc48a0dc64c23ddc584d5984228af148062593220caed5e0fdbca84e3fc0" exitCode=0
Oct 30 19:43:30.521497 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:30.521076    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41871c4d-4mpdm" event={"ID":"aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5","Type":"ContainerDied","Data":"70dbbc48a0dc64c23ddc584d5984228af148062593220caed5e0fdbca84e3fc0"}
Oct 30 19:43:31.696557 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:31.696526    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41871c4d-4mpdm"
Oct 30 19:43:31.747040 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:31.747009    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5-maas-api-xks-ca\") pod \"aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5\" (UID: \"aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5\") "
Oct 30 19:43:31.747232 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:31.747074    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5-maas-api-openshift-ca\") pod \"aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5\" (UID: \"aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5\") "
Oct 30 19:43:31.747232 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:31.747163    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5-kube-api-access-gjmx2\" (UniqueName: \"kubernetes.io/projected/aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5-kube-api-access-gjmx2\") pod \"aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5\" (UID: \"aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5\") "
Oct 30 19:43:31.747515 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:31.747491    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5-maas-api-openshift-ca" pod "aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5" (UID: "aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:31.748252 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:31.748231    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:31.751160 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:31.751133    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5-maas-api-xks-ca" pod "aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5" (UID: "aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:31.752989 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:31.752962    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5-kube-api-access-gjmx2" pod "aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5" (UID: "aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5"). InnerVolumeSpecName "kube-api-access-gjmx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:31.850113 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:31.850028    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:31.850113 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:31.850058    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjmx2\" (UniqueName: \"kubernetes.io/projected/aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5-kube-api-access-gjmx2\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:32.542316 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:32.542287    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41871c4d-4mpdm"
Oct 30 19:43:32.542497 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:32.542285    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41871c4d-4mpdm" event={"ID":"aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5","Type":"ContainerDied","Data":"a628c74692aad1f44928ad0131eee4ca1b1429223bb595ced24486bc1f176b14"}
Oct 30 19:43:32.542497 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:32.542409    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a628c74692aad1f44928ad0131eee4ca1b1429223bb595ced24486bc1f176b14"
Oct 30 19:43:34.518565 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:34.518532    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-579ddb-6756775486-ghzlj"]
Oct 30 19:43:34.540253 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:34.537648    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-579ddb-9b6756cf9-gfdfg"]
Oct 30 19:43:35.570119 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:35.570084    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1582090b-95fsw"]
Oct 30 19:43:35.570801 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:35.570786    5043 cpu_manager.go:414] "containerMap: removing container" podUID="cb491571-bc76-4a0b-b931-df9437d7a5bd" containerName="main"
Oct 30 19:43:35.570861 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:35.570803    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb491571-bc76-4a0b-b931-df9437d7a5bd" containerName="main"
Oct 30 19:43:35.570861 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:35.570831    5043 cpu_manager.go:414] "containerMap: removing container" podUID="aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5" containerName="revoke-keys"
Oct 30 19:43:35.570861 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:35.570836    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5" containerName="revoke-keys"
Oct 30 19:43:35.570963 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:35.570942    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5" containerName="revoke-keys"
Oct 30 19:43:35.570963 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:35.570959    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb491571-bc76-4a0b-b931-df9437d7a5bd" containerName="main"
Oct 30 19:43:35.574809 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:35.574791    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1582090b-95fsw"
Oct 30 19:43:35.579978 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:35.579952    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:43:35.580126 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:35.579987    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:43:35.580126 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:35.580082    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:43:35.580285 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:35.579968    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:43:35.585407 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:35.585376    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1582090b-95fsw"]
Oct 30 19:43:35.613319 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:35.613278    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27klt\" (UniqueName: \"kubernetes.io/projected/74855f09-5dc1-4dea-81e4-8dd04e20d7c0-kube-api-access-27klt\") pod \"maas-api-revoke-sub-1582090b-95fsw\" (UID: \"74855f09-5dc1-4dea-81e4-8dd04e20d7c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1582090b-95fsw"
Oct 30 19:43:35.613545 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:35.613443    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74855f09-5dc1-4dea-81e4-8dd04e20d7c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1582090b-95fsw\" (UID: \"74855f09-5dc1-4dea-81e4-8dd04e20d7c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1582090b-95fsw"
Oct 30 19:43:35.613545 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:35.613472    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74855f09-5dc1-4dea-81e4-8dd04e20d7c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1582090b-95fsw\" (UID: \"74855f09-5dc1-4dea-81e4-8dd04e20d7c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1582090b-95fsw"
Oct 30 19:43:35.714929 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:35.714892    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27klt\" (UniqueName: \"kubernetes.io/projected/74855f09-5dc1-4dea-81e4-8dd04e20d7c0-kube-api-access-27klt\") pod \"maas-api-revoke-sub-1582090b-95fsw\" (UID: \"74855f09-5dc1-4dea-81e4-8dd04e20d7c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1582090b-95fsw"
Oct 30 19:43:35.715129 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:35.715005    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74855f09-5dc1-4dea-81e4-8dd04e20d7c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1582090b-95fsw\" (UID: \"74855f09-5dc1-4dea-81e4-8dd04e20d7c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1582090b-95fsw"
Oct 30 19:43:35.715129 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:35.715034    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74855f09-5dc1-4dea-81e4-8dd04e20d7c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1582090b-95fsw\" (UID: \"74855f09-5dc1-4dea-81e4-8dd04e20d7c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1582090b-95fsw"
Oct 30 19:43:35.715635 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:35.715601    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74855f09-5dc1-4dea-81e4-8dd04e20d7c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1582090b-95fsw\" (UID: \"74855f09-5dc1-4dea-81e4-8dd04e20d7c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1582090b-95fsw"
Oct 30 19:43:35.717541 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:35.717518    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74855f09-5dc1-4dea-81e4-8dd04e20d7c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1582090b-95fsw\" (UID: \"74855f09-5dc1-4dea-81e4-8dd04e20d7c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1582090b-95fsw"
Oct 30 19:43:35.724302 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:35.724279    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27klt\" (UniqueName: \"kubernetes.io/projected/74855f09-5dc1-4dea-81e4-8dd04e20d7c0-kube-api-access-27klt\") pod \"maas-api-revoke-sub-1582090b-95fsw\" (UID: \"74855f09-5dc1-4dea-81e4-8dd04e20d7c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1582090b-95fsw"
Oct 30 19:43:35.893591 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:35.893499    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1582090b-95fsw"
Oct 30 19:43:36.179281 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:36.179254    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1582090b-95fsw"]
Oct 30 19:43:36.181016 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:43:36.180989    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74855f09_5dc1_4dea_81e4_8dd04e20d7c0.slice/crio-ff289df36a22c34717883b17cb0ec5717a1da3ef090413543264c31693396200 WatchSource:0}: Error finding container ff289df36a22c34717883b17cb0ec5717a1da3ef090413543264c31693396200: Status 404 returned error can't find the container with id ff289df36a22c34717883b17cb0ec5717a1da3ef090413543264c31693396200
Oct 30 19:43:36.601663 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:36.601635    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1582090b-95fsw" event={"ID":"74855f09-5dc1-4dea-81e4-8dd04e20d7c0","Type":"ContainerStarted","Data":"cceb9477ecf0e85c74e62bce8862c3dd04b6eb59d8fe30ce78fbb2ed05c37556"}
Oct 30 19:43:36.602033 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:36.601688    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1582090b-95fsw" event={"ID":"74855f09-5dc1-4dea-81e4-8dd04e20d7c0","Type":"ContainerStarted","Data":"ff289df36a22c34717883b17cb0ec5717a1da3ef090413543264c31693396200"}
Oct 30 19:43:36.622639 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:36.622597    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1582090b-95fsw" podStartSLOduration=1.6225827640000001 podStartE2EDuration="1.622582764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:36.620051106 +0000 UTC m=+1636.131111089" watchObservedRunningTime="2026-10-30 19:43:36.622582764 +0000 UTC m=+1636.133642728"
Oct 30 19:43:37.618345 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:37.618309    5043 generic.go:356] "Generic (PLEG): container finished" podID="74855f09-5dc1-4dea-81e4-8dd04e20d7c0" containerID="cceb9477ecf0e85c74e62bce8862c3dd04b6eb59d8fe30ce78fbb2ed05c37556" exitCode=0
Oct 30 19:43:37.618806 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:37.618389    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1582090b-95fsw" event={"ID":"74855f09-5dc1-4dea-81e4-8dd04e20d7c0","Type":"ContainerDied","Data":"cceb9477ecf0e85c74e62bce8862c3dd04b6eb59d8fe30ce78fbb2ed05c37556"}
Oct 30 19:43:38.820897 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:38.820867    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1582090b-95fsw"
Oct 30 19:43:38.859612 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:38.859568    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74855f09-5dc1-4dea-81e4-8dd04e20d7c0-kube-api-access-27klt\" (UniqueName: \"kubernetes.io/projected/74855f09-5dc1-4dea-81e4-8dd04e20d7c0-kube-api-access-27klt\") pod \"74855f09-5dc1-4dea-81e4-8dd04e20d7c0\" (UID: \"74855f09-5dc1-4dea-81e4-8dd04e20d7c0\") "
Oct 30 19:43:38.859863 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:38.859683    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74855f09-5dc1-4dea-81e4-8dd04e20d7c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74855f09-5dc1-4dea-81e4-8dd04e20d7c0-maas-api-xks-ca\") pod \"74855f09-5dc1-4dea-81e4-8dd04e20d7c0\" (UID: \"74855f09-5dc1-4dea-81e4-8dd04e20d7c0\") "
Oct 30 19:43:38.859956 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:38.859893    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74855f09-5dc1-4dea-81e4-8dd04e20d7c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74855f09-5dc1-4dea-81e4-8dd04e20d7c0-maas-api-openshift-ca\") pod \"74855f09-5dc1-4dea-81e4-8dd04e20d7c0\" (UID: \"74855f09-5dc1-4dea-81e4-8dd04e20d7c0\") "
Oct 30 19:43:38.861796 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:38.861558    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74855f09-5dc1-4dea-81e4-8dd04e20d7c0-maas-api-openshift-ca" pod "74855f09-5dc1-4dea-81e4-8dd04e20d7c0" (UID: "74855f09-5dc1-4dea-81e4-8dd04e20d7c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:38.896844 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:38.889839    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74855f09-5dc1-4dea-81e4-8dd04e20d7c0-maas-api-xks-ca" pod "74855f09-5dc1-4dea-81e4-8dd04e20d7c0" (UID: "74855f09-5dc1-4dea-81e4-8dd04e20d7c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:38.896844 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:38.889888    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74855f09-5dc1-4dea-81e4-8dd04e20d7c0-kube-api-access-27klt" pod "74855f09-5dc1-4dea-81e4-8dd04e20d7c0" (UID: "74855f09-5dc1-4dea-81e4-8dd04e20d7c0"). InnerVolumeSpecName "kube-api-access-27klt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:38.962215 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:38.962176    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27klt\" (UniqueName: \"kubernetes.io/projected/74855f09-5dc1-4dea-81e4-8dd04e20d7c0-kube-api-access-27klt\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:38.962215 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:38.962210    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74855f09-5dc1-4dea-81e4-8dd04e20d7c0-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:38.962215 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:38.962223    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74855f09-5dc1-4dea-81e4-8dd04e20d7c0-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:39.240013 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.239964    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-1e13df/test-model-1e13df-kserve-c8b994f5f-97xvv" podUID="4d3381d2-2f2e-4699-bd0b-300f2355df74" containerName="main" containerID="cri-o://4a1aaf8c7a924e98d7d08bde155a26cd5525bdb9e90bce1c19ba28e692dbdf7c" gracePeriod=45
Oct 30 19:43:39.417962 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.417921    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a66b56a0-mrbtw"]
Oct 30 19:43:39.419201 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.419170    5043 cpu_manager.go:414] "containerMap: removing container" podUID="74855f09-5dc1-4dea-81e4-8dd04e20d7c0" containerName="revoke-keys"
Oct 30 19:43:39.419368 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.419203    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74855f09-5dc1-4dea-81e4-8dd04e20d7c0" containerName="revoke-keys"
Oct 30 19:43:39.419414 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.419398    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="74855f09-5dc1-4dea-81e4-8dd04e20d7c0" containerName="revoke-keys"
Oct 30 19:43:39.423972 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.423940    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a66b56a0-mrbtw"
Oct 30 19:43:39.430604 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.430573    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a66b56a0-mrbtw"]
Oct 30 19:43:39.473138 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.473089    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11e9713c-21d8-495e-9ed4-a94017cb4f56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a66b56a0-mrbtw\" (UID: \"11e9713c-21d8-495e-9ed4-a94017cb4f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a66b56a0-mrbtw"
Oct 30 19:43:39.473615 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.473586    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx5b6\" (UniqueName: \"kubernetes.io/projected/11e9713c-21d8-495e-9ed4-a94017cb4f56-kube-api-access-xx5b6\") pod \"maas-api-revoke-sub-a66b56a0-mrbtw\" (UID: \"11e9713c-21d8-495e-9ed4-a94017cb4f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a66b56a0-mrbtw"
Oct 30 19:43:39.473918 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.473897    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11e9713c-21d8-495e-9ed4-a94017cb4f56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a66b56a0-mrbtw\" (UID: \"11e9713c-21d8-495e-9ed4-a94017cb4f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a66b56a0-mrbtw"
Oct 30 19:43:39.519628 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.519537    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-579ddb-6756775486-ghzlj" podUID="8861a11f-5436-41be-9c85-15a114f3b4cd" containerName="payload-processing" containerID="cri-o://912039d27946c685339eeb37352ba2124eff68cb08df6b9564323935cbe8d857" gracePeriod=25
Oct 30 19:43:39.538319 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.538275    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-579ddb-9b6756cf9-gfdfg" podUID="38d6d4ee-f2f5-4747-9e4d-c371701193ad" containerName="payload-pre-processing" containerID="cri-o://b63836170c1831649ad5c2f5df15a39a5a3788651b7f46138a2e545cc1c86746" gracePeriod=25
Oct 30 19:43:39.552963 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.552939    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1e13df/test-model-1e13df-kserve-c8b994f5f-97xvv"
Oct 30 19:43:39.591787 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.585880    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11e9713c-21d8-495e-9ed4-a94017cb4f56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a66b56a0-mrbtw\" (UID: \"11e9713c-21d8-495e-9ed4-a94017cb4f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a66b56a0-mrbtw"
Oct 30 19:43:39.591787 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.585979    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx5b6\" (UniqueName: \"kubernetes.io/projected/11e9713c-21d8-495e-9ed4-a94017cb4f56-kube-api-access-xx5b6\") pod \"maas-api-revoke-sub-a66b56a0-mrbtw\" (UID: \"11e9713c-21d8-495e-9ed4-a94017cb4f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a66b56a0-mrbtw"
Oct 30 19:43:39.591787 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.586033    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11e9713c-21d8-495e-9ed4-a94017cb4f56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a66b56a0-mrbtw\" (UID: \"11e9713c-21d8-495e-9ed4-a94017cb4f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a66b56a0-mrbtw"
Oct 30 19:43:39.591787 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.588506    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11e9713c-21d8-495e-9ed4-a94017cb4f56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a66b56a0-mrbtw\" (UID: \"11e9713c-21d8-495e-9ed4-a94017cb4f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a66b56a0-mrbtw"
Oct 30 19:43:39.591787 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.591609    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11e9713c-21d8-495e-9ed4-a94017cb4f56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a66b56a0-mrbtw\" (UID: \"11e9713c-21d8-495e-9ed4-a94017cb4f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a66b56a0-mrbtw"
Oct 30 19:43:39.599059 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.599024    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx5b6\" (UniqueName: \"kubernetes.io/projected/11e9713c-21d8-495e-9ed4-a94017cb4f56-kube-api-access-xx5b6\") pod \"maas-api-revoke-sub-a66b56a0-mrbtw\" (UID: \"11e9713c-21d8-495e-9ed4-a94017cb4f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a66b56a0-mrbtw"
Oct 30 19:43:39.639995 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.639963    5043 generic.go:356] "Generic (PLEG): container finished" podID="4d3381d2-2f2e-4699-bd0b-300f2355df74" containerID="4a1aaf8c7a924e98d7d08bde155a26cd5525bdb9e90bce1c19ba28e692dbdf7c" exitCode=0
Oct 30 19:43:39.640200 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.640007    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1e13df/test-model-1e13df-kserve-c8b994f5f-97xvv"
Oct 30 19:43:39.640200 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.640054    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1e13df/test-model-1e13df-kserve-c8b994f5f-97xvv" event={"ID":"4d3381d2-2f2e-4699-bd0b-300f2355df74","Type":"ContainerDied","Data":"4a1aaf8c7a924e98d7d08bde155a26cd5525bdb9e90bce1c19ba28e692dbdf7c"}
Oct 30 19:43:39.640200 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.640101    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1e13df/test-model-1e13df-kserve-c8b994f5f-97xvv" event={"ID":"4d3381d2-2f2e-4699-bd0b-300f2355df74","Type":"ContainerDied","Data":"dc62d98465e90920b04f81d391135a63869c76a2696f20d91e27d86be0363339"}
Oct 30 19:43:39.640200 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.640124    5043 scope.go:122] "RemoveContainer" containerID="4a1aaf8c7a924e98d7d08bde155a26cd5525bdb9e90bce1c19ba28e692dbdf7c"
Oct 30 19:43:39.642132 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.642113    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1582090b-95fsw"
Oct 30 19:43:39.642476 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.642373    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1582090b-95fsw" event={"ID":"74855f09-5dc1-4dea-81e4-8dd04e20d7c0","Type":"ContainerDied","Data":"ff289df36a22c34717883b17cb0ec5717a1da3ef090413543264c31693396200"}
Oct 30 19:43:39.642621 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.642605    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff289df36a22c34717883b17cb0ec5717a1da3ef090413543264c31693396200"
Oct 30 19:43:39.652365 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.652349    5043 scope.go:122] "RemoveContainer" containerID="4a1aaf8c7a924e98d7d08bde155a26cd5525bdb9e90bce1c19ba28e692dbdf7c"
Oct 30 19:43:39.652659 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:43:39.652640    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a1aaf8c7a924e98d7d08bde155a26cd5525bdb9e90bce1c19ba28e692dbdf7c\": container with ID starting with 4a1aaf8c7a924e98d7d08bde155a26cd5525bdb9e90bce1c19ba28e692dbdf7c not found: ID does not exist" containerID="4a1aaf8c7a924e98d7d08bde155a26cd5525bdb9e90bce1c19ba28e692dbdf7c"
Oct 30 19:43:39.652756 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.652664    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a1aaf8c7a924e98d7d08bde155a26cd5525bdb9e90bce1c19ba28e692dbdf7c"} err="failed to get container status \"4a1aaf8c7a924e98d7d08bde155a26cd5525bdb9e90bce1c19ba28e692dbdf7c\": rpc error: code = NotFound desc = could not find container \"4a1aaf8c7a924e98d7d08bde155a26cd5525bdb9e90bce1c19ba28e692dbdf7c\": container with ID starting with 4a1aaf8c7a924e98d7d08bde155a26cd5525bdb9e90bce1c19ba28e692dbdf7c not found: ID does not exist"
Oct 30 19:43:39.692971 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.691648    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d3381d2-2f2e-4699-bd0b-300f2355df74-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d3381d2-2f2e-4699-bd0b-300f2355df74-dshm\") pod \"4d3381d2-2f2e-4699-bd0b-300f2355df74\" (UID: \"4d3381d2-2f2e-4699-bd0b-300f2355df74\") "
Oct 30 19:43:39.692971 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.691707    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d3381d2-2f2e-4699-bd0b-300f2355df74-home\" (UniqueName: \"kubernetes.io/empty-dir/4d3381d2-2f2e-4699-bd0b-300f2355df74-home\") pod \"4d3381d2-2f2e-4699-bd0b-300f2355df74\" (UID: \"4d3381d2-2f2e-4699-bd0b-300f2355df74\") "
Oct 30 19:43:39.692971 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.691762    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d3381d2-2f2e-4699-bd0b-300f2355df74-tls-certs\" (UniqueName: \"kubernetes.io/secret/4d3381d2-2f2e-4699-bd0b-300f2355df74-tls-certs\") pod \"4d3381d2-2f2e-4699-bd0b-300f2355df74\" (UID: \"4d3381d2-2f2e-4699-bd0b-300f2355df74\") "
Oct 30 19:43:39.692971 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.691794    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d3381d2-2f2e-4699-bd0b-300f2355df74-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d3381d2-2f2e-4699-bd0b-300f2355df74-tmp-dir\") pod \"4d3381d2-2f2e-4699-bd0b-300f2355df74\" (UID: \"4d3381d2-2f2e-4699-bd0b-300f2355df74\") "
Oct 30 19:43:39.692971 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.691959    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d3381d2-2f2e-4699-bd0b-300f2355df74-kube-api-access-47lzd\" (UniqueName: \"kubernetes.io/projected/4d3381d2-2f2e-4699-bd0b-300f2355df74-kube-api-access-47lzd\") pod \"4d3381d2-2f2e-4699-bd0b-300f2355df74\" (UID: \"4d3381d2-2f2e-4699-bd0b-300f2355df74\") "
Oct 30 19:43:39.692971 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.692172    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d3381d2-2f2e-4699-bd0b-300f2355df74-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d3381d2-2f2e-4699-bd0b-300f2355df74-model-cache\") pod \"4d3381d2-2f2e-4699-bd0b-300f2355df74\" (UID: \"4d3381d2-2f2e-4699-bd0b-300f2355df74\") "
Oct 30 19:43:39.695653 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.695173    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d3381d2-2f2e-4699-bd0b-300f2355df74-model-cache" pod "4d3381d2-2f2e-4699-bd0b-300f2355df74" (UID: "4d3381d2-2f2e-4699-bd0b-300f2355df74"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:39.695653 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.695423    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d3381d2-2f2e-4699-bd0b-300f2355df74-tmp-dir" pod "4d3381d2-2f2e-4699-bd0b-300f2355df74" (UID: "4d3381d2-2f2e-4699-bd0b-300f2355df74"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:39.695653 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.695568    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d3381d2-2f2e-4699-bd0b-300f2355df74-home" pod "4d3381d2-2f2e-4699-bd0b-300f2355df74" (UID: "4d3381d2-2f2e-4699-bd0b-300f2355df74"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:39.699030 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.698998    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d3381d2-2f2e-4699-bd0b-300f2355df74-tls-certs" pod "4d3381d2-2f2e-4699-bd0b-300f2355df74" (UID: "4d3381d2-2f2e-4699-bd0b-300f2355df74"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:39.699212 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.699193    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d3381d2-2f2e-4699-bd0b-300f2355df74-dshm" pod "4d3381d2-2f2e-4699-bd0b-300f2355df74" (UID: "4d3381d2-2f2e-4699-bd0b-300f2355df74"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:39.700511 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.700346    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d3381d2-2f2e-4699-bd0b-300f2355df74-kube-api-access-47lzd" pod "4d3381d2-2f2e-4699-bd0b-300f2355df74" (UID: "4d3381d2-2f2e-4699-bd0b-300f2355df74"). InnerVolumeSpecName "kube-api-access-47lzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:39.746996 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.746966    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a66b56a0-mrbtw"
Oct 30 19:43:39.797062 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.794683    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47lzd\" (UniqueName: \"kubernetes.io/projected/4d3381d2-2f2e-4699-bd0b-300f2355df74-kube-api-access-47lzd\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:39.797062 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.794720    5043 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d3381d2-2f2e-4699-bd0b-300f2355df74-model-cache\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:39.797062 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.794783    5043 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d3381d2-2f2e-4699-bd0b-300f2355df74-dshm\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:39.797062 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.794797    5043 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d3381d2-2f2e-4699-bd0b-300f2355df74-home\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:39.797062 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.794813    5043 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d3381d2-2f2e-4699-bd0b-300f2355df74-tls-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:39.797062 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.794826    5043 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d3381d2-2f2e-4699-bd0b-300f2355df74-tmp-dir\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:39.966767 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.966721    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1e13df/test-model-1e13df-kserve-c8b994f5f-97xvv"]
Oct 30 19:43:39.970813 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:39.970789    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-1e13df/test-model-1e13df-kserve-c8b994f5f-97xvv"]
Oct 30 19:43:40.040775 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:40.040711    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a66b56a0-mrbtw"]
Oct 30 19:43:40.041159 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:43:40.041126    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11e9713c_21d8_495e_9ed4_a94017cb4f56.slice/crio-499313e63b464e2253fdc8164a90b76bc23d2d9399570c8dd90abc70815c3523 WatchSource:0}: Error finding container 499313e63b464e2253fdc8164a90b76bc23d2d9399570c8dd90abc70815c3523: Status 404 returned error can't find the container with id 499313e63b464e2253fdc8164a90b76bc23d2d9399570c8dd90abc70815c3523
Oct 30 19:43:40.647867 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:40.647787    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d3381d2-2f2e-4699-bd0b-300f2355df74" path="/var/lib/kubelet/pods/4d3381d2-2f2e-4699-bd0b-300f2355df74/volumes"
Oct 30 19:43:40.653390 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:40.653362    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a66b56a0-mrbtw" event={"ID":"11e9713c-21d8-495e-9ed4-a94017cb4f56","Type":"ContainerStarted","Data":"b28b22756fe6e624347d6edde315cd5cad6caea331500b4a317d92da1891528d"}
Oct 30 19:43:40.653535 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:40.653416    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a66b56a0-mrbtw" event={"ID":"11e9713c-21d8-495e-9ed4-a94017cb4f56","Type":"ContainerStarted","Data":"499313e63b464e2253fdc8164a90b76bc23d2d9399570c8dd90abc70815c3523"}
Oct 30 19:43:40.689187 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:40.689132    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a66b56a0-mrbtw" podStartSLOduration=1.689115465 podStartE2EDuration="1.689115465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:40.67636368 +0000 UTC m=+1640.187423650" watchObservedRunningTime="2026-10-30 19:43:40.689115465 +0000 UTC m=+1640.200175439"
Oct 30 19:43:41.666134 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:41.666097    5043 generic.go:356] "Generic (PLEG): container finished" podID="11e9713c-21d8-495e-9ed4-a94017cb4f56" containerID="b28b22756fe6e624347d6edde315cd5cad6caea331500b4a317d92da1891528d" exitCode=0
Oct 30 19:43:41.666549 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:41.666253    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a66b56a0-mrbtw" event={"ID":"11e9713c-21d8-495e-9ed4-a94017cb4f56","Type":"ContainerDied","Data":"b28b22756fe6e624347d6edde315cd5cad6caea331500b4a317d92da1891528d"}
Oct 30 19:43:42.836118 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:42.836070    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-579ddb-6qrtw"]
Oct 30 19:43:42.839653 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:42.839625    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-579ddb-6qrtw"]
Oct 30 19:43:43.010802 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:43.010775    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a66b56a0-mrbtw"
Oct 30 19:43:43.163805 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:43.163753    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11e9713c-21d8-495e-9ed4-a94017cb4f56-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11e9713c-21d8-495e-9ed4-a94017cb4f56-maas-api-xks-ca\") pod \"11e9713c-21d8-495e-9ed4-a94017cb4f56\" (UID: \"11e9713c-21d8-495e-9ed4-a94017cb4f56\") "
Oct 30 19:43:43.164048 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:43.164029    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11e9713c-21d8-495e-9ed4-a94017cb4f56-kube-api-access-xx5b6\" (UniqueName: \"kubernetes.io/projected/11e9713c-21d8-495e-9ed4-a94017cb4f56-kube-api-access-xx5b6\") pod \"11e9713c-21d8-495e-9ed4-a94017cb4f56\" (UID: \"11e9713c-21d8-495e-9ed4-a94017cb4f56\") "
Oct 30 19:43:43.164144 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:43.164066    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11e9713c-21d8-495e-9ed4-a94017cb4f56-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11e9713c-21d8-495e-9ed4-a94017cb4f56-maas-api-openshift-ca\") pod \"11e9713c-21d8-495e-9ed4-a94017cb4f56\" (UID: \"11e9713c-21d8-495e-9ed4-a94017cb4f56\") "
Oct 30 19:43:43.165003 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:43.164975    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11e9713c-21d8-495e-9ed4-a94017cb4f56-maas-api-openshift-ca" pod "11e9713c-21d8-495e-9ed4-a94017cb4f56" (UID: "11e9713c-21d8-495e-9ed4-a94017cb4f56"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:43.167715 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:43.167673    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11e9713c-21d8-495e-9ed4-a94017cb4f56-maas-api-xks-ca" pod "11e9713c-21d8-495e-9ed4-a94017cb4f56" (UID: "11e9713c-21d8-495e-9ed4-a94017cb4f56"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:43.178333 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:43.169062    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11e9713c-21d8-495e-9ed4-a94017cb4f56-kube-api-access-xx5b6" pod "11e9713c-21d8-495e-9ed4-a94017cb4f56" (UID: "11e9713c-21d8-495e-9ed4-a94017cb4f56"). InnerVolumeSpecName "kube-api-access-xx5b6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:43.266312 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:43.266180    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx5b6\" (UniqueName: \"kubernetes.io/projected/11e9713c-21d8-495e-9ed4-a94017cb4f56-kube-api-access-xx5b6\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:43.266312 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:43.266216    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11e9713c-21d8-495e-9ed4-a94017cb4f56-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:43.266312 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:43.266232    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11e9713c-21d8-495e-9ed4-a94017cb4f56-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:43.722666 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:43.722635    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a66b56a0-mrbtw"
Oct 30 19:43:43.722666 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:43.722659    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a66b56a0-mrbtw" event={"ID":"11e9713c-21d8-495e-9ed4-a94017cb4f56","Type":"ContainerDied","Data":"499313e63b464e2253fdc8164a90b76bc23d2d9399570c8dd90abc70815c3523"}
Oct 30 19:43:43.722938 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:43.722710    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="499313e63b464e2253fdc8164a90b76bc23d2d9399570c8dd90abc70815c3523"
Oct 30 19:43:44.060803 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:44.060699    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-579ddb-6756775486-ghzlj" podUID="8861a11f-5436-41be-9c85-15a114f3b4cd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:43:44.061497 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:44.061470    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-579ddb-9b6756cf9-gfdfg" podUID="38d6d4ee-f2f5-4747-9e4d-c371701193ad" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:43:44.647062 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:44.647031    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="412bfb73-ee32-4740-83aa-3dacd1a2355a" path="/var/lib/kubelet/pods/412bfb73-ee32-4740-83aa-3dacd1a2355a/volumes"
Oct 30 19:43:49.958159 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:49.958121    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8735990a-hnfjb"]
Oct 30 19:43:49.958843 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:49.958827    5043 cpu_manager.go:414] "containerMap: removing container" podUID="11e9713c-21d8-495e-9ed4-a94017cb4f56" containerName="revoke-keys"
Oct 30 19:43:49.958843 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:49.958843    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11e9713c-21d8-495e-9ed4-a94017cb4f56" containerName="revoke-keys"
Oct 30 19:43:49.958979 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:49.958866    5043 cpu_manager.go:414] "containerMap: removing container" podUID="4d3381d2-2f2e-4699-bd0b-300f2355df74" containerName="main"
Oct 30 19:43:49.958979 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:49.958872    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d3381d2-2f2e-4699-bd0b-300f2355df74" containerName="main"
Oct 30 19:43:49.959098 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:49.958988    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d3381d2-2f2e-4699-bd0b-300f2355df74" containerName="main"
Oct 30 19:43:49.959098 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:49.959004    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="11e9713c-21d8-495e-9ed4-a94017cb4f56" containerName="revoke-keys"
Oct 30 19:43:49.963087 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:49.963065    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8735990a-hnfjb"
Oct 30 19:43:49.972629 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:49.972609    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:43:49.972629 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:49.972621    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:43:49.981174 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:49.981147    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8735990a-hnfjb"]
Oct 30 19:43:49.991440 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:49.991415    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6x64\" (UniqueName: \"kubernetes.io/projected/bffdd089-54cb-40c3-8c7d-cc797874069b-kube-api-access-m6x64\") pod \"maas-api-revoke-sub-8735990a-hnfjb\" (UID: \"bffdd089-54cb-40c3-8c7d-cc797874069b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8735990a-hnfjb"
Oct 30 19:43:49.991580 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:49.991463    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bffdd089-54cb-40c3-8c7d-cc797874069b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8735990a-hnfjb\" (UID: \"bffdd089-54cb-40c3-8c7d-cc797874069b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8735990a-hnfjb"
Oct 30 19:43:49.991580 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:49.991520    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bffdd089-54cb-40c3-8c7d-cc797874069b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8735990a-hnfjb\" (UID: \"bffdd089-54cb-40c3-8c7d-cc797874069b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8735990a-hnfjb"
Oct 30 19:43:50.055416 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:50.055392    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:43:50.055672 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:50.055654    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:43:50.093103 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:50.093075    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6x64\" (UniqueName: \"kubernetes.io/projected/bffdd089-54cb-40c3-8c7d-cc797874069b-kube-api-access-m6x64\") pod \"maas-api-revoke-sub-8735990a-hnfjb\" (UID: \"bffdd089-54cb-40c3-8c7d-cc797874069b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8735990a-hnfjb"
Oct 30 19:43:50.093277 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:50.093124    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bffdd089-54cb-40c3-8c7d-cc797874069b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8735990a-hnfjb\" (UID: \"bffdd089-54cb-40c3-8c7d-cc797874069b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8735990a-hnfjb"
Oct 30 19:43:50.093277 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:50.093185    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bffdd089-54cb-40c3-8c7d-cc797874069b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8735990a-hnfjb\" (UID: \"bffdd089-54cb-40c3-8c7d-cc797874069b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8735990a-hnfjb"
Oct 30 19:43:50.094051 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:50.094021    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bffdd089-54cb-40c3-8c7d-cc797874069b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8735990a-hnfjb\" (UID: \"bffdd089-54cb-40c3-8c7d-cc797874069b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8735990a-hnfjb"
Oct 30 19:43:50.095791 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:50.095770    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bffdd089-54cb-40c3-8c7d-cc797874069b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8735990a-hnfjb\" (UID: \"bffdd089-54cb-40c3-8c7d-cc797874069b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8735990a-hnfjb"
Oct 30 19:43:50.105610 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:50.105578    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6x64\" (UniqueName: \"kubernetes.io/projected/bffdd089-54cb-40c3-8c7d-cc797874069b-kube-api-access-m6x64\") pod \"maas-api-revoke-sub-8735990a-hnfjb\" (UID: \"bffdd089-54cb-40c3-8c7d-cc797874069b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8735990a-hnfjb"
Oct 30 19:43:50.281083 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:50.281053    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8735990a-hnfjb"
Oct 30 19:43:50.582174 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:50.582143    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8735990a-hnfjb"]
Oct 30 19:43:50.583666 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:43:50.583638    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbffdd089_54cb_40c3_8c7d_cc797874069b.slice/crio-d6cad77efddd5c4b42656a101dabb9b43ab100325e68e0c7fbcf8ecba32793a8 WatchSource:0}: Error finding container d6cad77efddd5c4b42656a101dabb9b43ab100325e68e0c7fbcf8ecba32793a8: Status 404 returned error can't find the container with id d6cad77efddd5c4b42656a101dabb9b43ab100325e68e0c7fbcf8ecba32793a8
Oct 30 19:43:50.793962 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:50.793932    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8735990a-hnfjb" event={"ID":"bffdd089-54cb-40c3-8c7d-cc797874069b","Type":"ContainerStarted","Data":"d6cad77efddd5c4b42656a101dabb9b43ab100325e68e0c7fbcf8ecba32793a8"}
Oct 30 19:43:51.816409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:51.816084    5043 generic.go:356] "Generic (PLEG): container finished" podID="bffdd089-54cb-40c3-8c7d-cc797874069b" containerID="06f13c02498b60fc779f547f449d5caf499c83b45904191bd5db9798c0d3c7cd" exitCode=0
Oct 30 19:43:51.817099 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:51.817067    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8735990a-hnfjb" event={"ID":"bffdd089-54cb-40c3-8c7d-cc797874069b","Type":"ContainerDied","Data":"06f13c02498b60fc779f547f449d5caf499c83b45904191bd5db9798c0d3c7cd"}
Oct 30 19:43:53.025282 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:53.025250    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8735990a-hnfjb"
Oct 30 19:43:53.142367 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:53.142283    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bffdd089-54cb-40c3-8c7d-cc797874069b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bffdd089-54cb-40c3-8c7d-cc797874069b-maas-api-openshift-ca\") pod \"bffdd089-54cb-40c3-8c7d-cc797874069b\" (UID: \"bffdd089-54cb-40c3-8c7d-cc797874069b\") "
Oct 30 19:43:53.142546 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:53.142368    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bffdd089-54cb-40c3-8c7d-cc797874069b-kube-api-access-m6x64\" (UniqueName: \"kubernetes.io/projected/bffdd089-54cb-40c3-8c7d-cc797874069b-kube-api-access-m6x64\") pod \"bffdd089-54cb-40c3-8c7d-cc797874069b\" (UID: \"bffdd089-54cb-40c3-8c7d-cc797874069b\") "
Oct 30 19:43:53.142546 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:53.142420    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bffdd089-54cb-40c3-8c7d-cc797874069b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bffdd089-54cb-40c3-8c7d-cc797874069b-maas-api-xks-ca\") pod \"bffdd089-54cb-40c3-8c7d-cc797874069b\" (UID: \"bffdd089-54cb-40c3-8c7d-cc797874069b\") "
Oct 30 19:43:53.142697 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:53.142669    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bffdd089-54cb-40c3-8c7d-cc797874069b-maas-api-openshift-ca" pod "bffdd089-54cb-40c3-8c7d-cc797874069b" (UID: "bffdd089-54cb-40c3-8c7d-cc797874069b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:53.143222 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:53.143198    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bffdd089-54cb-40c3-8c7d-cc797874069b-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:53.144566 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:53.144544    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bffdd089-54cb-40c3-8c7d-cc797874069b-maas-api-xks-ca" pod "bffdd089-54cb-40c3-8c7d-cc797874069b" (UID: "bffdd089-54cb-40c3-8c7d-cc797874069b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:53.144648 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:53.144592    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bffdd089-54cb-40c3-8c7d-cc797874069b-kube-api-access-m6x64" pod "bffdd089-54cb-40c3-8c7d-cc797874069b" (UID: "bffdd089-54cb-40c3-8c7d-cc797874069b"). InnerVolumeSpecName "kube-api-access-m6x64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:53.244555 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:53.244521    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6x64\" (UniqueName: \"kubernetes.io/projected/bffdd089-54cb-40c3-8c7d-cc797874069b-kube-api-access-m6x64\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:53.244555 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:53.244552    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bffdd089-54cb-40c3-8c7d-cc797874069b-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:53.722281 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:53.722243    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb"]
Oct 30 19:43:53.722531 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:53.722502    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb" podUID="7c584699-3a9f-45e1-a380-cb724ff74820" containerName="istio-proxy" containerID="cri-o://7b4c6cf472d14e180bf7d8576147d493600fb998f62737c8a2636f6ff955e151" gracePeriod=30
Oct 30 19:43:53.927881 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:53.927594    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8735990a-hnfjb" event={"ID":"bffdd089-54cb-40c3-8c7d-cc797874069b","Type":"ContainerDied","Data":"d6cad77efddd5c4b42656a101dabb9b43ab100325e68e0c7fbcf8ecba32793a8"}
Oct 30 19:43:53.927881 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:53.927654    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6cad77efddd5c4b42656a101dabb9b43ab100325e68e0c7fbcf8ecba32793a8"
Oct 30 19:43:53.927881 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:53.927760    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8735990a-hnfjb"
Oct 30 19:43:54.061250 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:54.061160    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-579ddb-9b6756cf9-gfdfg" podUID="38d6d4ee-f2f5-4747-9e4d-c371701193ad" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:43:54.061631 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:54.061167    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-579ddb-6756775486-ghzlj" podUID="8861a11f-5436-41be-9c85-15a114f3b4cd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:43:54.242767 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:54.242713    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1e13df-tnfdg"]
Oct 30 19:43:54.243515 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:54.243493    5043 cpu_manager.go:414] "containerMap: removing container" podUID="bffdd089-54cb-40c3-8c7d-cc797874069b" containerName="revoke-keys"
Oct 30 19:43:54.243632 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:54.243517    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bffdd089-54cb-40c3-8c7d-cc797874069b" containerName="revoke-keys"
Oct 30 19:43:54.243727 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:54.243713    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="bffdd089-54cb-40c3-8c7d-cc797874069b" containerName="revoke-keys"
Oct 30 19:43:54.247570 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:54.247550    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1e13df-tnfdg"
Oct 30 19:43:54.253605 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:54.253589    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:43:54.253703 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:54.253646    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:43:54.253820 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:54.253748    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:43:54.255450 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:54.255434    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:43:54.262039 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:54.262019    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1e13df-tnfdg"]
Oct 30 19:43:54.368068 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:54.367985    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csnxl\" (UniqueName: \"kubernetes.io/projected/a1f56fc1-64a0-4781-994e-76445ed3a10c-kube-api-access-csnxl\") pod \"maas-api-revoke-keys-e2e-inf-b-1e13df-tnfdg\" (UID: \"a1f56fc1-64a0-4781-994e-76445ed3a10c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1e13df-tnfdg"
Oct 30 19:43:54.368068 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:54.368044    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1f56fc1-64a0-4781-994e-76445ed3a10c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1e13df-tnfdg\" (UID: \"a1f56fc1-64a0-4781-994e-76445ed3a10c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1e13df-tnfdg"
Oct 30 19:43:54.368281 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:54.368116    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1f56fc1-64a0-4781-994e-76445ed3a10c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1e13df-tnfdg\" (UID: \"a1f56fc1-64a0-4781-994e-76445ed3a10c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1e13df-tnfdg"
Oct 30 19:43:54.469977 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:54.469929    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1f56fc1-64a0-4781-994e-76445ed3a10c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1e13df-tnfdg\" (UID: \"a1f56fc1-64a0-4781-994e-76445ed3a10c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1e13df-tnfdg"
Oct 30 19:43:54.470271 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:54.470247    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csnxl\" (UniqueName: \"kubernetes.io/projected/a1f56fc1-64a0-4781-994e-76445ed3a10c-kube-api-access-csnxl\") pod \"maas-api-revoke-keys-e2e-inf-b-1e13df-tnfdg\" (UID: \"a1f56fc1-64a0-4781-994e-76445ed3a10c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1e13df-tnfdg"
Oct 30 19:43:54.470407 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:54.470304    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1f56fc1-64a0-4781-994e-76445ed3a10c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1e13df-tnfdg\" (UID: \"a1f56fc1-64a0-4781-994e-76445ed3a10c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1e13df-tnfdg"
Oct 30 19:43:54.470726 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:54.470698    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1f56fc1-64a0-4781-994e-76445ed3a10c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1e13df-tnfdg\" (UID: \"a1f56fc1-64a0-4781-994e-76445ed3a10c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1e13df-tnfdg"
Oct 30 19:43:54.479762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:54.473495    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1f56fc1-64a0-4781-994e-76445ed3a10c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1e13df-tnfdg\" (UID: \"a1f56fc1-64a0-4781-994e-76445ed3a10c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1e13df-tnfdg"
Oct 30 19:43:54.482314 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:54.482276    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csnxl\" (UniqueName: \"kubernetes.io/projected/a1f56fc1-64a0-4781-994e-76445ed3a10c-kube-api-access-csnxl\") pod \"maas-api-revoke-keys-e2e-inf-b-1e13df-tnfdg\" (UID: \"a1f56fc1-64a0-4781-994e-76445ed3a10c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1e13df-tnfdg"
Oct 30 19:43:54.565099 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:54.565071    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1e13df-tnfdg"
Oct 30 19:43:54.748311 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:54.748271    5043 patch_prober.go:29] interesting pod/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:43:54.748563 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:54.748352    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb" podUID="7c584699-3a9f-45e1-a380-cb724ff74820" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:43:54.859764 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:43:54.859718    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1f56fc1_64a0_4781_994e_76445ed3a10c.slice/crio-7fc7040cc1cbaa88c7cc0b6664caf6fe4372d888a10daf21574fef76117f4250 WatchSource:0}: Error finding container 7fc7040cc1cbaa88c7cc0b6664caf6fe4372d888a10daf21574fef76117f4250: Status 404 returned error can't find the container with id 7fc7040cc1cbaa88c7cc0b6664caf6fe4372d888a10daf21574fef76117f4250
Oct 30 19:43:54.860725 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:54.860498    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1e13df-tnfdg"]
Oct 30 19:43:54.936438 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:54.936407    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1e13df-tnfdg" event={"ID":"a1f56fc1-64a0-4781-994e-76445ed3a10c","Type":"ContainerStarted","Data":"7fc7040cc1cbaa88c7cc0b6664caf6fe4372d888a10daf21574fef76117f4250"}
Oct 30 19:43:54.977650 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:43:54.977611    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:43:55.041874 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:55.041840    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ac37c8/test-model-ac37c8-kserve-6f996db89c-xpd6c"]
Oct 30 19:43:55.260888 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:55.260861    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-744b57-t824w"]
Oct 30 19:43:55.265523 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:55.265502    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-744b57-t824w"
Oct 30 19:43:55.283387 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:55.283228    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-744b57-t824w"]
Oct 30 19:43:55.388431 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:55.388403    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8kjk\" (UniqueName: \"kubernetes.io/projected/7a41cf8f-43e9-434f-97a8-26fbab320e36-kube-api-access-c8kjk\") pod \"maas-api-revoke-keys-e2e-inf-a-744b57-t824w\" (UID: \"7a41cf8f-43e9-434f-97a8-26fbab320e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-744b57-t824w"
Oct 30 19:43:55.388604 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:55.388465    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a41cf8f-43e9-434f-97a8-26fbab320e36-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-744b57-t824w\" (UID: \"7a41cf8f-43e9-434f-97a8-26fbab320e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-744b57-t824w"
Oct 30 19:43:55.388692 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:55.388666    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a41cf8f-43e9-434f-97a8-26fbab320e36-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-744b57-t824w\" (UID: \"7a41cf8f-43e9-434f-97a8-26fbab320e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-744b57-t824w"
Oct 30 19:43:55.489896 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:55.489860    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a41cf8f-43e9-434f-97a8-26fbab320e36-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-744b57-t824w\" (UID: \"7a41cf8f-43e9-434f-97a8-26fbab320e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-744b57-t824w"
Oct 30 19:43:55.490099 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:55.490041    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8kjk\" (UniqueName: \"kubernetes.io/projected/7a41cf8f-43e9-434f-97a8-26fbab320e36-kube-api-access-c8kjk\") pod \"maas-api-revoke-keys-e2e-inf-a-744b57-t824w\" (UID: \"7a41cf8f-43e9-434f-97a8-26fbab320e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-744b57-t824w"
Oct 30 19:43:55.490174 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:55.490132    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a41cf8f-43e9-434f-97a8-26fbab320e36-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-744b57-t824w\" (UID: \"7a41cf8f-43e9-434f-97a8-26fbab320e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-744b57-t824w"
Oct 30 19:43:55.490827 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:55.490798    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a41cf8f-43e9-434f-97a8-26fbab320e36-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-744b57-t824w\" (UID: \"7a41cf8f-43e9-434f-97a8-26fbab320e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-744b57-t824w"
Oct 30 19:43:55.493008 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:55.492982    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a41cf8f-43e9-434f-97a8-26fbab320e36-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-744b57-t824w\" (UID: \"7a41cf8f-43e9-434f-97a8-26fbab320e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-744b57-t824w"
Oct 30 19:43:55.507122 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:55.507096    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8kjk\" (UniqueName: \"kubernetes.io/projected/7a41cf8f-43e9-434f-97a8-26fbab320e36-kube-api-access-c8kjk\") pod \"maas-api-revoke-keys-e2e-inf-a-744b57-t824w\" (UID: \"7a41cf8f-43e9-434f-97a8-26fbab320e36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-744b57-t824w"
Oct 30 19:43:55.587209 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:55.587138    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-744b57-t824w"
Oct 30 19:43:55.879183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:55.879156    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-744b57-t824w"]
Oct 30 19:43:55.901949 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:43:55.901906    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a41cf8f_43e9_434f_97a8_26fbab320e36.slice/crio-37cfdd2e96e232ab5b81833fbed5b90e87379c62d7178aaa469bd9bcd716845d WatchSource:0}: Error finding container 37cfdd2e96e232ab5b81833fbed5b90e87379c62d7178aaa469bd9bcd716845d: Status 404 returned error can't find the container with id 37cfdd2e96e232ab5b81833fbed5b90e87379c62d7178aaa469bd9bcd716845d
Oct 30 19:43:55.945878 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:55.945850    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-744b57-t824w" event={"ID":"7a41cf8f-43e9-434f-97a8-26fbab320e36","Type":"ContainerStarted","Data":"37cfdd2e96e232ab5b81833fbed5b90e87379c62d7178aaa469bd9bcd716845d"}
Oct 30 19:43:55.947595 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:55.947570    5043 generic.go:356] "Generic (PLEG): container finished" podID="a1f56fc1-64a0-4781-994e-76445ed3a10c" containerID="8ba2e53999da8deb64dc6b9eeb950178d9298c6f9da9438878ed15c81c326e11" exitCode=0
Oct 30 19:43:55.947713 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:55.947628    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1e13df-tnfdg" event={"ID":"a1f56fc1-64a0-4781-994e-76445ed3a10c","Type":"ContainerDied","Data":"8ba2e53999da8deb64dc6b9eeb950178d9298c6f9da9438878ed15c81c326e11"}
Oct 30 19:43:56.957492 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:56.957458    5043 generic.go:356] "Generic (PLEG): container finished" podID="7a41cf8f-43e9-434f-97a8-26fbab320e36" containerID="1bf504fdbd6b15f9cc35713ca798432f9cc87c2226ba368fd3b6ed5ead8d5efd" exitCode=0
Oct 30 19:43:56.957957 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:56.957537    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-744b57-t824w" event={"ID":"7a41cf8f-43e9-434f-97a8-26fbab320e36","Type":"ContainerDied","Data":"1bf504fdbd6b15f9cc35713ca798432f9cc87c2226ba368fd3b6ed5ead8d5efd"}
Oct 30 19:43:57.169433 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:57.169396    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1e13df-tnfdg"
Oct 30 19:43:57.218824 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:57.218793    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1f56fc1-64a0-4781-994e-76445ed3a10c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1f56fc1-64a0-4781-994e-76445ed3a10c-maas-api-openshift-ca\") pod \"a1f56fc1-64a0-4781-994e-76445ed3a10c\" (UID: \"a1f56fc1-64a0-4781-994e-76445ed3a10c\") "
Oct 30 19:43:57.219195 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:57.219173    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1f56fc1-64a0-4781-994e-76445ed3a10c-kube-api-access-csnxl\" (UniqueName: \"kubernetes.io/projected/a1f56fc1-64a0-4781-994e-76445ed3a10c-kube-api-access-csnxl\") pod \"a1f56fc1-64a0-4781-994e-76445ed3a10c\" (UID: \"a1f56fc1-64a0-4781-994e-76445ed3a10c\") "
Oct 30 19:43:57.219311 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:57.219219    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1f56fc1-64a0-4781-994e-76445ed3a10c-maas-api-openshift-ca" pod "a1f56fc1-64a0-4781-994e-76445ed3a10c" (UID: "a1f56fc1-64a0-4781-994e-76445ed3a10c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:57.219311 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:57.219249    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1f56fc1-64a0-4781-994e-76445ed3a10c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1f56fc1-64a0-4781-994e-76445ed3a10c-maas-api-xks-ca\") pod \"a1f56fc1-64a0-4781-994e-76445ed3a10c\" (UID: \"a1f56fc1-64a0-4781-994e-76445ed3a10c\") "
Oct 30 19:43:57.219934 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:57.219906    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1f56fc1-64a0-4781-994e-76445ed3a10c-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:57.222192 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:57.222163    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1f56fc1-64a0-4781-994e-76445ed3a10c-maas-api-xks-ca" pod "a1f56fc1-64a0-4781-994e-76445ed3a10c" (UID: "a1f56fc1-64a0-4781-994e-76445ed3a10c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:57.222290 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:57.222232    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1f56fc1-64a0-4781-994e-76445ed3a10c-kube-api-access-csnxl" pod "a1f56fc1-64a0-4781-994e-76445ed3a10c" (UID: "a1f56fc1-64a0-4781-994e-76445ed3a10c"). InnerVolumeSpecName "kube-api-access-csnxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:57.321465 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:57.321379    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csnxl\" (UniqueName: \"kubernetes.io/projected/a1f56fc1-64a0-4781-994e-76445ed3a10c-kube-api-access-csnxl\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:57.321465 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:57.321411    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1f56fc1-64a0-4781-994e-76445ed3a10c-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:57.968517 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:57.968483    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1e13df-tnfdg"
Oct 30 19:43:57.968517 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:57.968497    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1e13df-tnfdg" event={"ID":"a1f56fc1-64a0-4781-994e-76445ed3a10c","Type":"ContainerDied","Data":"7fc7040cc1cbaa88c7cc0b6664caf6fe4372d888a10daf21574fef76117f4250"}
Oct 30 19:43:57.969102 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:57.968591    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fc7040cc1cbaa88c7cc0b6664caf6fe4372d888a10daf21574fef76117f4250"
Oct 30 19:43:58.238950 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:58.238878    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-744b57-t824w"
Oct 30 19:43:58.337135 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:58.337099    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a41cf8f-43e9-434f-97a8-26fbab320e36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a41cf8f-43e9-434f-97a8-26fbab320e36-maas-api-xks-ca\") pod \"7a41cf8f-43e9-434f-97a8-26fbab320e36\" (UID: \"7a41cf8f-43e9-434f-97a8-26fbab320e36\") "
Oct 30 19:43:58.337339 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:58.337226    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a41cf8f-43e9-434f-97a8-26fbab320e36-kube-api-access-c8kjk\" (UniqueName: \"kubernetes.io/projected/7a41cf8f-43e9-434f-97a8-26fbab320e36-kube-api-access-c8kjk\") pod \"7a41cf8f-43e9-434f-97a8-26fbab320e36\" (UID: \"7a41cf8f-43e9-434f-97a8-26fbab320e36\") "
Oct 30 19:43:58.337476 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:58.337460    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a41cf8f-43e9-434f-97a8-26fbab320e36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a41cf8f-43e9-434f-97a8-26fbab320e36-maas-api-openshift-ca\") pod \"7a41cf8f-43e9-434f-97a8-26fbab320e36\" (UID: \"7a41cf8f-43e9-434f-97a8-26fbab320e36\") "
Oct 30 19:43:58.337808 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:58.337781    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a41cf8f-43e9-434f-97a8-26fbab320e36-maas-api-openshift-ca" pod "7a41cf8f-43e9-434f-97a8-26fbab320e36" (UID: "7a41cf8f-43e9-434f-97a8-26fbab320e36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:58.338103 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:58.338086    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a41cf8f-43e9-434f-97a8-26fbab320e36-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:58.339649 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:58.339622    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a41cf8f-43e9-434f-97a8-26fbab320e36-maas-api-xks-ca" pod "7a41cf8f-43e9-434f-97a8-26fbab320e36" (UID: "7a41cf8f-43e9-434f-97a8-26fbab320e36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:58.339902 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:58.339886    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a41cf8f-43e9-434f-97a8-26fbab320e36-kube-api-access-c8kjk" pod "7a41cf8f-43e9-434f-97a8-26fbab320e36" (UID: "7a41cf8f-43e9-434f-97a8-26fbab320e36"). InnerVolumeSpecName "kube-api-access-c8kjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:58.439843 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:58.439807    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a41cf8f-43e9-434f-97a8-26fbab320e36-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:58.439843 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:58.439840    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8kjk\" (UniqueName: \"kubernetes.io/projected/7a41cf8f-43e9-434f-97a8-26fbab320e36-kube-api-access-c8kjk\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:59.054774 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:59.054096    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-744b57-t824w" event={"ID":"7a41cf8f-43e9-434f-97a8-26fbab320e36","Type":"ContainerDied","Data":"37cfdd2e96e232ab5b81833fbed5b90e87379c62d7178aaa469bd9bcd716845d"}
Oct 30 19:43:59.054774 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:59.054173    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37cfdd2e96e232ab5b81833fbed5b90e87379c62d7178aaa469bd9bcd716845d"
Oct 30 19:43:59.054774 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:59.054245    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-744b57-t824w"
Oct 30 19:43:59.063036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:59.062989    5043 generic.go:356] "Generic (PLEG): container finished" podID="7c584699-3a9f-45e1-a380-cb724ff74820" containerID="7b4c6cf472d14e180bf7d8576147d493600fb998f62737c8a2636f6ff955e151" exitCode=0
Oct 30 19:43:59.063205 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:59.063031    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb" event={"ID":"7c584699-3a9f-45e1-a380-cb724ff74820","Type":"ContainerDied","Data":"7b4c6cf472d14e180bf7d8576147d493600fb998f62737c8a2636f6ff955e151"}
Oct 30 19:43:59.063205 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:59.063110    5043 scope.go:122] "RemoveContainer" containerID="0ca07795e91bd72eaa34df2f3bb17a19d451880cb6df0d83257b6302915182b7"
Oct 30 19:43:59.108426 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:59.108400    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb"
Oct 30 19:43:59.251215 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:59.251178    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c584699-3a9f-45e1-a380-cb724ff74820-istio-token\" (UniqueName: \"kubernetes.io/projected/7c584699-3a9f-45e1-a380-cb724ff74820-istio-token\") pod \"7c584699-3a9f-45e1-a380-cb724ff74820\" (UID: \"7c584699-3a9f-45e1-a380-cb724ff74820\") "
Oct 30 19:43:59.251423 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:59.251247    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c584699-3a9f-45e1-a380-cb724ff74820-kube-api-access-4vw4c\" (UniqueName: \"kubernetes.io/projected/7c584699-3a9f-45e1-a380-cb724ff74820-kube-api-access-4vw4c\") pod \"7c584699-3a9f-45e1-a380-cb724ff74820\" (UID: \"7c584699-3a9f-45e1-a380-cb724ff74820\") "
Oct 30 19:43:59.251423 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:59.251293    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c584699-3a9f-45e1-a380-cb724ff74820-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c584699-3a9f-45e1-a380-cb724ff74820-credential-socket\") pod \"7c584699-3a9f-45e1-a380-cb724ff74820\" (UID: \"7c584699-3a9f-45e1-a380-cb724ff74820\") "
Oct 30 19:43:59.251423 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:59.251357    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c584699-3a9f-45e1-a380-cb724ff74820-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c584699-3a9f-45e1-a380-cb724ff74820-istio-data\") pod \"7c584699-3a9f-45e1-a380-cb724ff74820\" (UID: \"7c584699-3a9f-45e1-a380-cb724ff74820\") "
Oct 30 19:43:59.251593 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:59.251535    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c584699-3a9f-45e1-a380-cb724ff74820-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c584699-3a9f-45e1-a380-cb724ff74820-workload-certs\") pod \"7c584699-3a9f-45e1-a380-cb724ff74820\" (UID: \"7c584699-3a9f-45e1-a380-cb724ff74820\") "
Oct 30 19:43:59.251593 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:59.251575    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c584699-3a9f-45e1-a380-cb724ff74820-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c584699-3a9f-45e1-a380-cb724ff74820-istiod-ca-cert\") pod \"7c584699-3a9f-45e1-a380-cb724ff74820\" (UID: \"7c584699-3a9f-45e1-a380-cb724ff74820\") "
Oct 30 19:43:59.251694 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:59.251641    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c584699-3a9f-45e1-a380-cb724ff74820-credential-socket" pod "7c584699-3a9f-45e1-a380-cb724ff74820" (UID: "7c584699-3a9f-45e1-a380-cb724ff74820"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:59.251777 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:59.251686    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c584699-3a9f-45e1-a380-cb724ff74820-istio-data" pod "7c584699-3a9f-45e1-a380-cb724ff74820" (UID: "7c584699-3a9f-45e1-a380-cb724ff74820"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:59.251841 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:59.251785    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c584699-3a9f-45e1-a380-cb724ff74820-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c584699-3a9f-45e1-a380-cb724ff74820-workload-socket\") pod \"7c584699-3a9f-45e1-a380-cb724ff74820\" (UID: \"7c584699-3a9f-45e1-a380-cb724ff74820\") "
Oct 30 19:43:59.251897 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:59.251852    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c584699-3a9f-45e1-a380-cb724ff74820-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c584699-3a9f-45e1-a380-cb724ff74820-istio-envoy\") pod \"7c584699-3a9f-45e1-a380-cb724ff74820\" (UID: \"7c584699-3a9f-45e1-a380-cb724ff74820\") "
Oct 30 19:43:59.251967 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:59.251896    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c584699-3a9f-45e1-a380-cb724ff74820-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c584699-3a9f-45e1-a380-cb724ff74820-istio-podinfo\") pod \"7c584699-3a9f-45e1-a380-cb724ff74820\" (UID: \"7c584699-3a9f-45e1-a380-cb724ff74820\") "
Oct 30 19:43:59.251967 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:59.251894    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c584699-3a9f-45e1-a380-cb724ff74820-workload-certs" pod "7c584699-3a9f-45e1-a380-cb724ff74820" (UID: "7c584699-3a9f-45e1-a380-cb724ff74820"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:59.252087 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:59.252057    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c584699-3a9f-45e1-a380-cb724ff74820-istiod-ca-cert" pod "7c584699-3a9f-45e1-a380-cb724ff74820" (UID: "7c584699-3a9f-45e1-a380-cb724ff74820"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:59.252482 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:59.252300    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c584699-3a9f-45e1-a380-cb724ff74820-workload-socket" pod "7c584699-3a9f-45e1-a380-cb724ff74820" (UID: "7c584699-3a9f-45e1-a380-cb724ff74820"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:59.252604 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:59.252586    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c584699-3a9f-45e1-a380-cb724ff74820-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:59.252675 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:59.252606    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c584699-3a9f-45e1-a380-cb724ff74820-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:59.252675 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:59.252622    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c584699-3a9f-45e1-a380-cb724ff74820-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:59.252675 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:59.252637    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c584699-3a9f-45e1-a380-cb724ff74820-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:59.252675 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:59.252649    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c584699-3a9f-45e1-a380-cb724ff74820-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:59.253884 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:59.253857    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c584699-3a9f-45e1-a380-cb724ff74820-istio-token" pod "7c584699-3a9f-45e1-a380-cb724ff74820" (UID: "7c584699-3a9f-45e1-a380-cb724ff74820"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:59.254100 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:59.254079    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c584699-3a9f-45e1-a380-cb724ff74820-kube-api-access-4vw4c" pod "7c584699-3a9f-45e1-a380-cb724ff74820" (UID: "7c584699-3a9f-45e1-a380-cb724ff74820"). InnerVolumeSpecName "kube-api-access-4vw4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:59.254952 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:59.254928    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c584699-3a9f-45e1-a380-cb724ff74820-istio-envoy" pod "7c584699-3a9f-45e1-a380-cb724ff74820" (UID: "7c584699-3a9f-45e1-a380-cb724ff74820"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:59.254952 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:59.254938    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c584699-3a9f-45e1-a380-cb724ff74820-istio-podinfo" pod "7c584699-3a9f-45e1-a380-cb724ff74820" (UID: "7c584699-3a9f-45e1-a380-cb724ff74820"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:43:59.353954 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:59.353921    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c584699-3a9f-45e1-a380-cb724ff74820-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:59.353954 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:59.353948    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c584699-3a9f-45e1-a380-cb724ff74820-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:59.353954 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:59.353961    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c584699-3a9f-45e1-a380-cb724ff74820-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:59.354185 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:59.353974    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vw4c\" (UniqueName: \"kubernetes.io/projected/7c584699-3a9f-45e1-a380-cb724ff74820-kube-api-access-4vw4c\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:59.859410 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:59.859382    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-579ddb-6756775486-ghzlj"
Oct 30 19:43:59.897997 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:59.897972    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-579ddb-9b6756cf9-gfdfg"
Oct 30 19:43:59.962431 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:59.962397    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38d6d4ee-f2f5-4747-9e4d-c371701193ad-kube-api-access-4d595\" (UniqueName: \"kubernetes.io/projected/38d6d4ee-f2f5-4747-9e4d-c371701193ad-kube-api-access-4d595\") pod \"38d6d4ee-f2f5-4747-9e4d-c371701193ad\" (UID: \"38d6d4ee-f2f5-4747-9e4d-c371701193ad\") "
Oct 30 19:43:59.962622 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:59.962549    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38d6d4ee-f2f5-4747-9e4d-c371701193ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38d6d4ee-f2f5-4747-9e4d-c371701193ad-plugins-config-volume\") pod \"38d6d4ee-f2f5-4747-9e4d-c371701193ad\" (UID: \"38d6d4ee-f2f5-4747-9e4d-c371701193ad\") "
Oct 30 19:43:59.962751 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:59.962717    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8861a11f-5436-41be-9c85-15a114f3b4cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8861a11f-5436-41be-9c85-15a114f3b4cd-plugins-config-volume\") pod \"8861a11f-5436-41be-9c85-15a114f3b4cd\" (UID: \"8861a11f-5436-41be-9c85-15a114f3b4cd\") "
Oct 30 19:43:59.962829 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:59.962798    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8861a11f-5436-41be-9c85-15a114f3b4cd-kube-api-access-x4wdm\" (UniqueName: \"kubernetes.io/projected/8861a11f-5436-41be-9c85-15a114f3b4cd-kube-api-access-x4wdm\") pod \"8861a11f-5436-41be-9c85-15a114f3b4cd\" (UID: \"8861a11f-5436-41be-9c85-15a114f3b4cd\") "
Oct 30 19:43:59.963168 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:59.963130    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8861a11f-5436-41be-9c85-15a114f3b4cd-plugins-config-volume" pod "8861a11f-5436-41be-9c85-15a114f3b4cd" (UID: "8861a11f-5436-41be-9c85-15a114f3b4cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:59.963480 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:59.963456    5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8861a11f-5436-41be-9c85-15a114f3b4cd-plugins-config-volume\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:59.964818 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:59.964785    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38d6d4ee-f2f5-4747-9e4d-c371701193ad-plugins-config-volume" pod "38d6d4ee-f2f5-4747-9e4d-c371701193ad" (UID: "38d6d4ee-f2f5-4747-9e4d-c371701193ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:59.966174 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:59.966145    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8861a11f-5436-41be-9c85-15a114f3b4cd-kube-api-access-x4wdm" pod "8861a11f-5436-41be-9c85-15a114f3b4cd" (UID: "8861a11f-5436-41be-9c85-15a114f3b4cd"). InnerVolumeSpecName "kube-api-access-x4wdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:59.967214 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:43:59.967183    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38d6d4ee-f2f5-4747-9e4d-c371701193ad-kube-api-access-4d595" pod "38d6d4ee-f2f5-4747-9e4d-c371701193ad" (UID: "38d6d4ee-f2f5-4747-9e4d-c371701193ad"). InnerVolumeSpecName "kube-api-access-4d595". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:00.065855 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:00.065710    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4wdm\" (UniqueName: \"kubernetes.io/projected/8861a11f-5436-41be-9c85-15a114f3b4cd-kube-api-access-x4wdm\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:00.065855 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:00.065755    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d595\" (UniqueName: \"kubernetes.io/projected/38d6d4ee-f2f5-4747-9e4d-c371701193ad-kube-api-access-4d595\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:00.065855 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:00.065770    5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38d6d4ee-f2f5-4747-9e4d-c371701193ad-plugins-config-volume\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:00.074644 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:00.074617    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb"
Oct 30 19:44:00.074837 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:00.074639    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb" event={"ID":"7c584699-3a9f-45e1-a380-cb724ff74820","Type":"ContainerDied","Data":"e88470840cb59d88b988de45e15f4233527e1621b82cd4b53bc2e1f57af24e5b"}
Oct 30 19:44:00.074837 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:00.074715    5043 scope.go:122] "RemoveContainer" containerID="7b4c6cf472d14e180bf7d8576147d493600fb998f62737c8a2636f6ff955e151"
Oct 30 19:44:00.076570 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:00.076546    5043 generic.go:356] "Generic (PLEG): container finished" podID="38d6d4ee-f2f5-4747-9e4d-c371701193ad" containerID="b63836170c1831649ad5c2f5df15a39a5a3788651b7f46138a2e545cc1c86746" exitCode=0
Oct 30 19:44:00.076698 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:00.076587    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-579ddb-9b6756cf9-gfdfg"
Oct 30 19:44:00.076698 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:00.076636    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-579ddb-9b6756cf9-gfdfg" event={"ID":"38d6d4ee-f2f5-4747-9e4d-c371701193ad","Type":"ContainerDied","Data":"b63836170c1831649ad5c2f5df15a39a5a3788651b7f46138a2e545cc1c86746"}
Oct 30 19:44:00.076826 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:00.076700    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-579ddb-9b6756cf9-gfdfg" event={"ID":"38d6d4ee-f2f5-4747-9e4d-c371701193ad","Type":"ContainerDied","Data":"7e8aab8797664798d7fb89242917c866b8498bddca8008e7c0601bfd52127852"}
Oct 30 19:44:00.078335 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:00.078311    5043 generic.go:356] "Generic (PLEG): container finished" podID="8861a11f-5436-41be-9c85-15a114f3b4cd" containerID="912039d27946c685339eeb37352ba2124eff68cb08df6b9564323935cbe8d857" exitCode=0
Oct 30 19:44:00.078565 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:00.078550    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-579ddb-6756775486-ghzlj" event={"ID":"8861a11f-5436-41be-9c85-15a114f3b4cd","Type":"ContainerDied","Data":"912039d27946c685339eeb37352ba2124eff68cb08df6b9564323935cbe8d857"}
Oct 30 19:44:00.078637 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:00.078621    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-579ddb-6756775486-ghzlj" event={"ID":"8861a11f-5436-41be-9c85-15a114f3b4cd","Type":"ContainerDied","Data":"aacf17b9dbc65bfe283cfd33a5d8196eef8b246d2ad79706c4e076a293abfce8"}
Oct 30 19:44:00.078698 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:00.078685    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-579ddb-6756775486-ghzlj"
Oct 30 19:44:00.090548 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:00.090530    5043 scope.go:122] "RemoveContainer" containerID="b63836170c1831649ad5c2f5df15a39a5a3788651b7f46138a2e545cc1c86746"
Oct 30 19:44:00.101258 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:00.101239    5043 scope.go:122] "RemoveContainer" containerID="b63836170c1831649ad5c2f5df15a39a5a3788651b7f46138a2e545cc1c86746"
Oct 30 19:44:00.101531 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:44:00.101509    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b63836170c1831649ad5c2f5df15a39a5a3788651b7f46138a2e545cc1c86746\": container with ID starting with b63836170c1831649ad5c2f5df15a39a5a3788651b7f46138a2e545cc1c86746 not found: ID does not exist" containerID="b63836170c1831649ad5c2f5df15a39a5a3788651b7f46138a2e545cc1c86746"
Oct 30 19:44:00.101601 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:00.101543    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b63836170c1831649ad5c2f5df15a39a5a3788651b7f46138a2e545cc1c86746"} err="failed to get container status \"b63836170c1831649ad5c2f5df15a39a5a3788651b7f46138a2e545cc1c86746\": rpc error: code = NotFound desc = could not find container \"b63836170c1831649ad5c2f5df15a39a5a3788651b7f46138a2e545cc1c86746\": container with ID starting with b63836170c1831649ad5c2f5df15a39a5a3788651b7f46138a2e545cc1c86746 not found: ID does not exist"
Oct 30 19:44:00.101601 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:00.101560    5043 scope.go:122] "RemoveContainer" containerID="912039d27946c685339eeb37352ba2124eff68cb08df6b9564323935cbe8d857"
Oct 30 19:44:00.118881 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:00.114564    5043 scope.go:122] "RemoveContainer" containerID="912039d27946c685339eeb37352ba2124eff68cb08df6b9564323935cbe8d857"
Oct 30 19:44:00.118881 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:44:00.114931    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"912039d27946c685339eeb37352ba2124eff68cb08df6b9564323935cbe8d857\": container with ID starting with 912039d27946c685339eeb37352ba2124eff68cb08df6b9564323935cbe8d857 not found: ID does not exist" containerID="912039d27946c685339eeb37352ba2124eff68cb08df6b9564323935cbe8d857"
Oct 30 19:44:00.118881 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:00.114970    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"912039d27946c685339eeb37352ba2124eff68cb08df6b9564323935cbe8d857"} err="failed to get container status \"912039d27946c685339eeb37352ba2124eff68cb08df6b9564323935cbe8d857\": rpc error: code = NotFound desc = could not find container \"912039d27946c685339eeb37352ba2124eff68cb08df6b9564323935cbe8d857\": container with ID starting with 912039d27946c685339eeb37352ba2124eff68cb08df6b9564323935cbe8d857 not found: ID does not exist"
Oct 30 19:44:00.138277 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:00.138205    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb"]
Oct 30 19:44:00.148413 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:00.148384    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-579ddb-openshift-default-677f4b6845-mwmnb"]
Oct 30 19:44:00.187135 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:00.185984    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-579ddb-6756775486-ghzlj"]
Oct 30 19:44:00.202468 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:00.202430    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-579ddb-6756775486-ghzlj"]
Oct 30 19:44:00.210880 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:00.210849    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-579ddb-9b6756cf9-gfdfg"]
Oct 30 19:44:00.221231 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:00.221203    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-579ddb-9b6756cf9-gfdfg"]
Oct 30 19:44:00.649472 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:00.649443    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38d6d4ee-f2f5-4747-9e4d-c371701193ad" path="/var/lib/kubelet/pods/38d6d4ee-f2f5-4747-9e4d-c371701193ad/volumes"
Oct 30 19:44:00.650140 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:00.650112    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c584699-3a9f-45e1-a380-cb724ff74820" path="/var/lib/kubelet/pods/7c584699-3a9f-45e1-a380-cb724ff74820/volumes"
Oct 30 19:44:00.650634 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:00.650609    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8861a11f-5436-41be-9c85-15a114f3b4cd" path="/var/lib/kubelet/pods/8861a11f-5436-41be-9c85-15a114f3b4cd/volumes"
Oct 30 19:44:04.256119 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:04.256072    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89a3f6-6gmm5"]
Oct 30 19:44:04.256820 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:04.256799    5043 cpu_manager.go:414] "containerMap: removing container" podUID="a1f56fc1-64a0-4781-994e-76445ed3a10c" containerName="revoke-keys"
Oct 30 19:44:04.256820 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:04.256821    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1f56fc1-64a0-4781-994e-76445ed3a10c" containerName="revoke-keys"
Oct 30 19:44:04.256977 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:04.256844    5043 cpu_manager.go:414] "containerMap: removing container" podUID="38d6d4ee-f2f5-4747-9e4d-c371701193ad" containerName="payload-pre-processing"
Oct 30 19:44:04.256977 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:04.256851    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38d6d4ee-f2f5-4747-9e4d-c371701193ad" containerName="payload-pre-processing"
Oct 30 19:44:04.256977 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:04.256875    5043 cpu_manager.go:414] "containerMap: removing container" podUID="7c584699-3a9f-45e1-a380-cb724ff74820" containerName="istio-proxy"
Oct 30 19:44:04.256977 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:04.256880    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c584699-3a9f-45e1-a380-cb724ff74820" containerName="istio-proxy"
Oct 30 19:44:04.256977 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:04.256889    5043 cpu_manager.go:414] "containerMap: removing container" podUID="7a41cf8f-43e9-434f-97a8-26fbab320e36" containerName="revoke-keys"
Oct 30 19:44:04.256977 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:04.256897    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a41cf8f-43e9-434f-97a8-26fbab320e36" containerName="revoke-keys"
Oct 30 19:44:04.256977 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:04.256917    5043 cpu_manager.go:414] "containerMap: removing container" podUID="8861a11f-5436-41be-9c85-15a114f3b4cd" containerName="payload-processing"
Oct 30 19:44:04.256977 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:04.256924    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8861a11f-5436-41be-9c85-15a114f3b4cd" containerName="payload-processing"
Oct 30 19:44:04.257224 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:04.257154    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c584699-3a9f-45e1-a380-cb724ff74820" containerName="istio-proxy"
Oct 30 19:44:04.257224 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:04.257166    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a41cf8f-43e9-434f-97a8-26fbab320e36" containerName="revoke-keys"
Oct 30 19:44:04.257224 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:04.257175    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="8861a11f-5436-41be-9c85-15a114f3b4cd" containerName="payload-processing"
Oct 30 19:44:04.257224 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:04.257186    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c584699-3a9f-45e1-a380-cb724ff74820" containerName="istio-proxy"
Oct 30 19:44:04.257224 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:04.257194    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1f56fc1-64a0-4781-994e-76445ed3a10c" containerName="revoke-keys"
Oct 30 19:44:04.257224 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:04.257201    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="38d6d4ee-f2f5-4747-9e4d-c371701193ad" containerName="payload-pre-processing"
Oct 30 19:44:04.262768 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:04.262711    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89a3f6-6gmm5"
Oct 30 19:44:04.274577 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:04.274033    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89a3f6-6gmm5"]
Oct 30 19:44:04.289487 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:04.277960    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:44:04.289487 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:04.278049    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:44:04.289487 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:04.278464    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:44:04.289487 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:04.278501    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:44:04.346196 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:04.346166    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ab2f697-94f3-4b6a-928c-c6ef38b81c14-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-89a3f6-6gmm5\" (UID: \"1ab2f697-94f3-4b6a-928c-c6ef38b81c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89a3f6-6gmm5"
Oct 30 19:44:04.346196 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:04.346198    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfdlp\" (UniqueName: \"kubernetes.io/projected/1ab2f697-94f3-4b6a-928c-c6ef38b81c14-kube-api-access-qfdlp\") pod \"maas-api-revoke-keys-e2e-inf-a-89a3f6-6gmm5\" (UID: \"1ab2f697-94f3-4b6a-928c-c6ef38b81c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89a3f6-6gmm5"
Oct 30 19:44:04.346439 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:04.346421    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ab2f697-94f3-4b6a-928c-c6ef38b81c14-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-89a3f6-6gmm5\" (UID: \"1ab2f697-94f3-4b6a-928c-c6ef38b81c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89a3f6-6gmm5"
Oct 30 19:44:04.447521 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:04.447487    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ab2f697-94f3-4b6a-928c-c6ef38b81c14-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-89a3f6-6gmm5\" (UID: \"1ab2f697-94f3-4b6a-928c-c6ef38b81c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89a3f6-6gmm5"
Oct 30 19:44:04.447697 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:04.447623    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ab2f697-94f3-4b6a-928c-c6ef38b81c14-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-89a3f6-6gmm5\" (UID: \"1ab2f697-94f3-4b6a-928c-c6ef38b81c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89a3f6-6gmm5"
Oct 30 19:44:04.447697 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:04.447645    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfdlp\" (UniqueName: \"kubernetes.io/projected/1ab2f697-94f3-4b6a-928c-c6ef38b81c14-kube-api-access-qfdlp\") pod \"maas-api-revoke-keys-e2e-inf-a-89a3f6-6gmm5\" (UID: \"1ab2f697-94f3-4b6a-928c-c6ef38b81c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89a3f6-6gmm5"
Oct 30 19:44:04.448159 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:04.448134    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ab2f697-94f3-4b6a-928c-c6ef38b81c14-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-89a3f6-6gmm5\" (UID: \"1ab2f697-94f3-4b6a-928c-c6ef38b81c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89a3f6-6gmm5"
Oct 30 19:44:04.450172 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:04.450156    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ab2f697-94f3-4b6a-928c-c6ef38b81c14-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-89a3f6-6gmm5\" (UID: \"1ab2f697-94f3-4b6a-928c-c6ef38b81c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89a3f6-6gmm5"
Oct 30 19:44:04.497945 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:04.493375    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfdlp\" (UniqueName: \"kubernetes.io/projected/1ab2f697-94f3-4b6a-928c-c6ef38b81c14-kube-api-access-qfdlp\") pod \"maas-api-revoke-keys-e2e-inf-a-89a3f6-6gmm5\" (UID: \"1ab2f697-94f3-4b6a-928c-c6ef38b81c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89a3f6-6gmm5"
Oct 30 19:44:04.498593 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:04.498565    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1e13df-78b7ccf4bc-8ldn2"]
Oct 30 19:44:04.591873 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:04.591793    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89a3f6-6gmm5"
Oct 30 19:44:04.897111 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:04.897085    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89a3f6-6gmm5"]
Oct 30 19:44:04.898803 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:44:04.898779    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ab2f697_94f3_4b6a_928c_c6ef38b81c14.slice/crio-6fd5d5e11d1a2f8460cc99a4603868c845a79a8cb2e8db652591ecc0d0fca4f7 WatchSource:0}: Error finding container 6fd5d5e11d1a2f8460cc99a4603868c845a79a8cb2e8db652591ecc0d0fca4f7: Status 404 returned error can't find the container with id 6fd5d5e11d1a2f8460cc99a4603868c845a79a8cb2e8db652591ecc0d0fca4f7
Oct 30 19:44:05.182834 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:05.179900    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89a3f6-6gmm5" event={"ID":"1ab2f697-94f3-4b6a-928c-c6ef38b81c14","Type":"ContainerStarted","Data":"6fd5d5e11d1a2f8460cc99a4603868c845a79a8cb2e8db652591ecc0d0fca4f7"}
Oct 30 19:44:05.471203 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:05.471176    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-744b57-597d6c8986-4v2zh"]
Oct 30 19:44:05.488589 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:05.488559    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-744b57-77cc66bfc9-2fcz9"]
Oct 30 19:44:06.191917 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:06.191876    5043 generic.go:356] "Generic (PLEG): container finished" podID="1ab2f697-94f3-4b6a-928c-c6ef38b81c14" containerID="fd4aa340d89f4d52f3f2dafd088d63d2eb016152599dab59b6bb5c4592a1c3ae" exitCode=0
Oct 30 19:44:06.192112 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:06.191924    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89a3f6-6gmm5" event={"ID":"1ab2f697-94f3-4b6a-928c-c6ef38b81c14","Type":"ContainerDied","Data":"fd4aa340d89f4d52f3f2dafd088d63d2eb016152599dab59b6bb5c4592a1c3ae"}
Oct 30 19:44:06.669821 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:06.669794    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-744b57-t824w"]
Oct 30 19:44:06.675948 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:06.675920    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-744b57-t824w"]
Oct 30 19:44:07.390233 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:07.390202    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89a3f6-6gmm5"
Oct 30 19:44:07.530186 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:07.530153    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ab2f697-94f3-4b6a-928c-c6ef38b81c14-kube-api-access-qfdlp\" (UniqueName: \"kubernetes.io/projected/1ab2f697-94f3-4b6a-928c-c6ef38b81c14-kube-api-access-qfdlp\") pod \"1ab2f697-94f3-4b6a-928c-c6ef38b81c14\" (UID: \"1ab2f697-94f3-4b6a-928c-c6ef38b81c14\") "
Oct 30 19:44:07.530368 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:07.530268    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ab2f697-94f3-4b6a-928c-c6ef38b81c14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ab2f697-94f3-4b6a-928c-c6ef38b81c14-maas-api-xks-ca\") pod \"1ab2f697-94f3-4b6a-928c-c6ef38b81c14\" (UID: \"1ab2f697-94f3-4b6a-928c-c6ef38b81c14\") "
Oct 30 19:44:07.530368 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:07.530337    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ab2f697-94f3-4b6a-928c-c6ef38b81c14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ab2f697-94f3-4b6a-928c-c6ef38b81c14-maas-api-openshift-ca\") pod \"1ab2f697-94f3-4b6a-928c-c6ef38b81c14\" (UID: \"1ab2f697-94f3-4b6a-928c-c6ef38b81c14\") "
Oct 30 19:44:07.530706 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:07.530681    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ab2f697-94f3-4b6a-928c-c6ef38b81c14-maas-api-openshift-ca" pod "1ab2f697-94f3-4b6a-928c-c6ef38b81c14" (UID: "1ab2f697-94f3-4b6a-928c-c6ef38b81c14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:07.531176 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:07.531157    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ab2f697-94f3-4b6a-928c-c6ef38b81c14-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:07.532489 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:07.532463    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ab2f697-94f3-4b6a-928c-c6ef38b81c14-maas-api-xks-ca" pod "1ab2f697-94f3-4b6a-928c-c6ef38b81c14" (UID: "1ab2f697-94f3-4b6a-928c-c6ef38b81c14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:44:07.532592 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:07.532517    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ab2f697-94f3-4b6a-928c-c6ef38b81c14-kube-api-access-qfdlp" pod "1ab2f697-94f3-4b6a-928c-c6ef38b81c14" (UID: "1ab2f697-94f3-4b6a-928c-c6ef38b81c14"). InnerVolumeSpecName "kube-api-access-qfdlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:07.632503 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:07.632460    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ab2f697-94f3-4b6a-928c-c6ef38b81c14-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:07.632651 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:07.632517    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfdlp\" (UniqueName: \"kubernetes.io/projected/1ab2f697-94f3-4b6a-928c-c6ef38b81c14-kube-api-access-qfdlp\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:08.209217 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:08.209187    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89a3f6-6gmm5"
Oct 30 19:44:08.209217 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:08.209213    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89a3f6-6gmm5" event={"ID":"1ab2f697-94f3-4b6a-928c-c6ef38b81c14","Type":"ContainerDied","Data":"6fd5d5e11d1a2f8460cc99a4603868c845a79a8cb2e8db652591ecc0d0fca4f7"}
Oct 30 19:44:08.209667 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:08.209256    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fd5d5e11d1a2f8460cc99a4603868c845a79a8cb2e8db652591ecc0d0fca4f7"
Oct 30 19:44:08.646542 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:08.646439    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a41cf8f-43e9-434f-97a8-26fbab320e36" path="/var/lib/kubelet/pods/7a41cf8f-43e9-434f-97a8-26fbab320e36/volumes"
Oct 30 19:44:09.499545 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:09.499500    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1e13df-78b7ccf4bc-8ldn2" podUID="3cd1e9e6-6a23-40d8-aa51-0266d9c61446" containerName="payload-pre-processing" containerID="cri-o://a214ad1cb247e0cbbae5d35dd948109e53bdc09f70b4d3c45c691db8f7ec9d41" gracePeriod=25
Oct 30 19:44:10.066859 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:10.066819    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ac37c8/test-model-ac37c8-kserve-6f996db89c-xpd6c" podUID="79d37674-5d73-4b92-a57f-66bee5d7574c" containerName="main" containerID="cri-o://3381e9be570960018305c822463ee7a9d58d291982c99948ca87a22753ae60ba" gracePeriod=45
Oct 30 19:44:10.228398 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:10.228361    5043 generic.go:356] "Generic (PLEG): container finished" podID="79d37674-5d73-4b92-a57f-66bee5d7574c" containerID="3381e9be570960018305c822463ee7a9d58d291982c99948ca87a22753ae60ba" exitCode=0
Oct 30 19:44:10.228573 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:10.228446    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ac37c8/test-model-ac37c8-kserve-6f996db89c-xpd6c" event={"ID":"79d37674-5d73-4b92-a57f-66bee5d7574c","Type":"ContainerDied","Data":"3381e9be570960018305c822463ee7a9d58d291982c99948ca87a22753ae60ba"}
Oct 30 19:44:10.355217 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:10.355191    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ac37c8/test-model-ac37c8-kserve-6f996db89c-xpd6c"
Oct 30 19:44:10.367663 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:10.367636    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79d37674-5d73-4b92-a57f-66bee5d7574c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/79d37674-5d73-4b92-a57f-66bee5d7574c-dshm\") pod \"79d37674-5d73-4b92-a57f-66bee5d7574c\" (UID: \"79d37674-5d73-4b92-a57f-66bee5d7574c\") "
Oct 30 19:44:10.367845 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:10.367675    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79d37674-5d73-4b92-a57f-66bee5d7574c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79d37674-5d73-4b92-a57f-66bee5d7574c-model-cache\") pod \"79d37674-5d73-4b92-a57f-66bee5d7574c\" (UID: \"79d37674-5d73-4b92-a57f-66bee5d7574c\") "
Oct 30 19:44:10.367845 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:10.367705    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79d37674-5d73-4b92-a57f-66bee5d7574c-tls-certs\" (UniqueName: \"kubernetes.io/secret/79d37674-5d73-4b92-a57f-66bee5d7574c-tls-certs\") pod \"79d37674-5d73-4b92-a57f-66bee5d7574c\" (UID: \"79d37674-5d73-4b92-a57f-66bee5d7574c\") "
Oct 30 19:44:10.367845 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:10.367828    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79d37674-5d73-4b92-a57f-66bee5d7574c-kube-api-access-ll7rc\" (UniqueName: \"kubernetes.io/projected/79d37674-5d73-4b92-a57f-66bee5d7574c-kube-api-access-ll7rc\") pod \"79d37674-5d73-4b92-a57f-66bee5d7574c\" (UID: \"79d37674-5d73-4b92-a57f-66bee5d7574c\") "
Oct 30 19:44:10.368027 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:10.367938    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79d37674-5d73-4b92-a57f-66bee5d7574c-model-cache" pod "79d37674-5d73-4b92-a57f-66bee5d7574c" (UID: "79d37674-5d73-4b92-a57f-66bee5d7574c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:10.368027 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:10.367967    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79d37674-5d73-4b92-a57f-66bee5d7574c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79d37674-5d73-4b92-a57f-66bee5d7574c-tmp-dir\") pod \"79d37674-5d73-4b92-a57f-66bee5d7574c\" (UID: \"79d37674-5d73-4b92-a57f-66bee5d7574c\") "
Oct 30 19:44:10.368027 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:10.367993    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79d37674-5d73-4b92-a57f-66bee5d7574c-home\" (UniqueName: \"kubernetes.io/empty-dir/79d37674-5d73-4b92-a57f-66bee5d7574c-home\") pod \"79d37674-5d73-4b92-a57f-66bee5d7574c\" (UID: \"79d37674-5d73-4b92-a57f-66bee5d7574c\") "
Oct 30 19:44:10.368228 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:10.368201    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79d37674-5d73-4b92-a57f-66bee5d7574c-tmp-dir" pod "79d37674-5d73-4b92-a57f-66bee5d7574c" (UID: "79d37674-5d73-4b92-a57f-66bee5d7574c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:10.368464 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:10.368317    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79d37674-5d73-4b92-a57f-66bee5d7574c-home" pod "79d37674-5d73-4b92-a57f-66bee5d7574c" (UID: "79d37674-5d73-4b92-a57f-66bee5d7574c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:10.368669 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:10.368648    5043 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79d37674-5d73-4b92-a57f-66bee5d7574c-tmp-dir\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:10.368754 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:10.368674    5043 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79d37674-5d73-4b92-a57f-66bee5d7574c-home\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:10.368754 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:10.368689    5043 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79d37674-5d73-4b92-a57f-66bee5d7574c-model-cache\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:10.370271 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:10.370213    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79d37674-5d73-4b92-a57f-66bee5d7574c-tls-certs" pod "79d37674-5d73-4b92-a57f-66bee5d7574c" (UID: "79d37674-5d73-4b92-a57f-66bee5d7574c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:44:10.370356 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:10.370295    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79d37674-5d73-4b92-a57f-66bee5d7574c-dshm" pod "79d37674-5d73-4b92-a57f-66bee5d7574c" (UID: "79d37674-5d73-4b92-a57f-66bee5d7574c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:10.370356 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:10.370305    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79d37674-5d73-4b92-a57f-66bee5d7574c-kube-api-access-ll7rc" pod "79d37674-5d73-4b92-a57f-66bee5d7574c" (UID: "79d37674-5d73-4b92-a57f-66bee5d7574c"). InnerVolumeSpecName "kube-api-access-ll7rc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:10.470023 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:10.469989    5043 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79d37674-5d73-4b92-a57f-66bee5d7574c-dshm\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:10.470023 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:10.470021    5043 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79d37674-5d73-4b92-a57f-66bee5d7574c-tls-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:10.470237 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:10.470039    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll7rc\" (UniqueName: \"kubernetes.io/projected/79d37674-5d73-4b92-a57f-66bee5d7574c-kube-api-access-ll7rc\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:10.472431 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:10.472375    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-744b57-597d6c8986-4v2zh" podUID="42404324-3c5e-4842-abad-eeb53637016e" containerName="payload-processing" containerID="cri-o://763c854283902c6ac04af0d967742004a31067845c7da719362e06ba090a1132" gracePeriod=25
Oct 30 19:44:10.488999 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:10.488963    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-744b57-77cc66bfc9-2fcz9" podUID="4ef41111-49f3-4fd9-9587-2a1934f50797" containerName="payload-pre-processing" containerID="cri-o://af0969d1dcc550e1c71ac468872d2556fec54bb81e7596638a935a164561936c" gracePeriod=25
Oct 30 19:44:10.828476 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:10.828449    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-744b57-597d6c8986-4v2zh"
Oct 30 19:44:10.932540 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:10.932455    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42404324-3c5e-4842-abad-eeb53637016e-kube-api-access-k9k9m\" (UniqueName: \"kubernetes.io/projected/42404324-3c5e-4842-abad-eeb53637016e-kube-api-access-k9k9m\") pod \"42404324-3c5e-4842-abad-eeb53637016e\" (UID: \"42404324-3c5e-4842-abad-eeb53637016e\") "
Oct 30 19:44:10.932708 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:10.932588    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42404324-3c5e-4842-abad-eeb53637016e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42404324-3c5e-4842-abad-eeb53637016e-plugins-config-volume\") pod \"42404324-3c5e-4842-abad-eeb53637016e\" (UID: \"42404324-3c5e-4842-abad-eeb53637016e\") "
Oct 30 19:44:10.933077 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:10.933045    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42404324-3c5e-4842-abad-eeb53637016e-plugins-config-volume" pod "42404324-3c5e-4842-abad-eeb53637016e" (UID: "42404324-3c5e-4842-abad-eeb53637016e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:10.933418 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:10.933397    5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42404324-3c5e-4842-abad-eeb53637016e-plugins-config-volume\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:10.934956 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:10.934935    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42404324-3c5e-4842-abad-eeb53637016e-kube-api-access-k9k9m" pod "42404324-3c5e-4842-abad-eeb53637016e" (UID: "42404324-3c5e-4842-abad-eeb53637016e"). InnerVolumeSpecName "kube-api-access-k9k9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:11.035231 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:11.035193    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9k9m\" (UniqueName: \"kubernetes.io/projected/42404324-3c5e-4842-abad-eeb53637016e-kube-api-access-k9k9m\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:11.239276 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:11.239236    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ac37c8/test-model-ac37c8-kserve-6f996db89c-xpd6c" event={"ID":"79d37674-5d73-4b92-a57f-66bee5d7574c","Type":"ContainerDied","Data":"fd577e752e56a6dd17e418eb407a332bff08662d060e7c32d0222577344b157f"}
Oct 30 19:44:11.239438 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:11.239295    5043 scope.go:122] "RemoveContainer" containerID="3381e9be570960018305c822463ee7a9d58d291982c99948ca87a22753ae60ba"
Oct 30 19:44:11.239480 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:11.239457    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ac37c8/test-model-ac37c8-kserve-6f996db89c-xpd6c"
Oct 30 19:44:11.242534 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:11.242458    5043 generic.go:356] "Generic (PLEG): container finished" podID="42404324-3c5e-4842-abad-eeb53637016e" containerID="763c854283902c6ac04af0d967742004a31067845c7da719362e06ba090a1132" exitCode=0
Oct 30 19:44:11.242660 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:11.242631    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-744b57-597d6c8986-4v2zh"
Oct 30 19:44:11.242754 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:11.242682    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-744b57-597d6c8986-4v2zh" event={"ID":"42404324-3c5e-4842-abad-eeb53637016e","Type":"ContainerDied","Data":"763c854283902c6ac04af0d967742004a31067845c7da719362e06ba090a1132"}
Oct 30 19:44:11.242817 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:11.242770    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-744b57-597d6c8986-4v2zh" event={"ID":"42404324-3c5e-4842-abad-eeb53637016e","Type":"ContainerDied","Data":"8ebc712d937210811483acb7644a8c493fa2d787e7e2b41307322c7a3c1e9c8e"}
Oct 30 19:44:11.254980 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:11.254958    5043 scope.go:122] "RemoveContainer" containerID="763c854283902c6ac04af0d967742004a31067845c7da719362e06ba090a1132"
Oct 30 19:44:11.265920 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:11.265897    5043 scope.go:122] "RemoveContainer" containerID="763c854283902c6ac04af0d967742004a31067845c7da719362e06ba090a1132"
Oct 30 19:44:11.266200 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:44:11.266176    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"763c854283902c6ac04af0d967742004a31067845c7da719362e06ba090a1132\": container with ID starting with 763c854283902c6ac04af0d967742004a31067845c7da719362e06ba090a1132 not found: ID does not exist" containerID="763c854283902c6ac04af0d967742004a31067845c7da719362e06ba090a1132"
Oct 30 19:44:11.266296 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:11.266212    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"763c854283902c6ac04af0d967742004a31067845c7da719362e06ba090a1132"} err="failed to get container status \"763c854283902c6ac04af0d967742004a31067845c7da719362e06ba090a1132\": rpc error: code = NotFound desc = could not find container \"763c854283902c6ac04af0d967742004a31067845c7da719362e06ba090a1132\": container with ID starting with 763c854283902c6ac04af0d967742004a31067845c7da719362e06ba090a1132 not found: ID does not exist"
Oct 30 19:44:11.295424 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:11.295394    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ac37c8/test-model-ac37c8-kserve-6f996db89c-xpd6c"]
Oct 30 19:44:11.304747 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:11.304710    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ac37c8/test-model-ac37c8-kserve-6f996db89c-xpd6c"]
Oct 30 19:44:11.335752 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:11.335683    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-744b57-597d6c8986-4v2zh"]
Oct 30 19:44:11.349776 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:11.349708    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-744b57-597d6c8986-4v2zh"]
Oct 30 19:44:12.659386 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:12.659343    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42404324-3c5e-4842-abad-eeb53637016e" path="/var/lib/kubelet/pods/42404324-3c5e-4842-abad-eeb53637016e/volumes"
Oct 30 19:44:12.659942 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:12.659914    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79d37674-5d73-4b92-a57f-66bee5d7574c" path="/var/lib/kubelet/pods/79d37674-5d73-4b92-a57f-66bee5d7574c/volumes"
Oct 30 19:44:13.910873 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:13.910838    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-744b57-77cc66bfc9-2fcz9" podUID="4ef41111-49f3-4fd9-9587-2a1934f50797" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:44:13.952638 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:13.952606    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc"]
Oct 30 19:44:13.952840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:13.952820    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc" podUID="53142e30-10c0-413a-ae61-fb1b29f5fd10" containerName="istio-proxy" containerID="cri-o://a3b8b859589480824dd49f3184380eb6d291f261a3f0e93a16262fc4419e998f" gracePeriod=30
Oct 30 19:44:13.997557 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:13.997516    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf"]
Oct 30 19:44:13.998078 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:13.998012    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf" podUID="c5c180ba-4a6f-4fb5-9976-1bd4b3526081" containerName="istio-proxy" containerID="cri-o://5479dc4f6f1bee401de9e545375dbdc8b111f8d453118eb9c4e84b4535fb8a02" gracePeriod=30
Oct 30 19:44:14.491887 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:14.491853    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-89a3f6-db46ffd87-8l4s5"]
Oct 30 19:44:14.527680 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:14.527634    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1e13df-78b7ccf4bc-8ldn2" podUID="3cd1e9e6-6a23-40d8-aa51-0266d9c61446" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:44:16.364759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:16.363990    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96"]
Oct 30 19:44:16.364759 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:16.364233    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96" podUID="377f3c78-650d-4680-aa0f-0373eba791eb" containerName="istio-proxy" containerID="cri-o://7f2abe2aab6157719288b52cb9c9cf58ac9724e3b060f8c88bf92e623d1784fa" gracePeriod=30
Oct 30 19:44:16.564402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:16.564373    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1e13df-tnfdg"]
Oct 30 19:44:16.571208 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:16.571177    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1e13df-tnfdg"]
Oct 30 19:44:16.645931 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:16.645848    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1f56fc1-64a0-4781-994e-76445ed3a10c" path="/var/lib/kubelet/pods/a1f56fc1-64a0-4781-994e-76445ed3a10c/volumes"
Oct 30 19:44:17.811332 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:17.811285    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd"]
Oct 30 19:44:17.811776 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:17.811527    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd" podUID="5ff1923b-6547-4e9b-8810-3f79648e3272" containerName="istio-proxy" containerID="cri-o://d4061dda458306dde91ff8f2a254ce9bb14709f6ea844c758630cff33d1aa8a4" gracePeriod=30
Oct 30 19:44:18.299033 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:18.299001    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ac37c8-fqfbn"]
Oct 30 19:44:18.299662 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:18.299651    5043 cpu_manager.go:414] "containerMap: removing container" podUID="7c584699-3a9f-45e1-a380-cb724ff74820" containerName="istio-proxy"
Oct 30 19:44:18.299723 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:18.299663    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c584699-3a9f-45e1-a380-cb724ff74820" containerName="istio-proxy"
Oct 30 19:44:18.299723 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:18.299677    5043 cpu_manager.go:414] "containerMap: removing container" podUID="1ab2f697-94f3-4b6a-928c-c6ef38b81c14" containerName="revoke-keys"
Oct 30 19:44:18.299723 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:18.299682    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ab2f697-94f3-4b6a-928c-c6ef38b81c14" containerName="revoke-keys"
Oct 30 19:44:18.299723 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:18.299691    5043 cpu_manager.go:414] "containerMap: removing container" podUID="79d37674-5d73-4b92-a57f-66bee5d7574c" containerName="main"
Oct 30 19:44:18.299723 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:18.299697    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79d37674-5d73-4b92-a57f-66bee5d7574c" containerName="main"
Oct 30 19:44:18.299723 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:18.299713    5043 cpu_manager.go:414] "containerMap: removing container" podUID="42404324-3c5e-4842-abad-eeb53637016e" containerName="payload-processing"
Oct 30 19:44:18.299723 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:18.299717    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42404324-3c5e-4842-abad-eeb53637016e" containerName="payload-processing"
Oct 30 19:44:18.299970 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:18.299833    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="42404324-3c5e-4842-abad-eeb53637016e" containerName="payload-processing"
Oct 30 19:44:18.299970 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:18.299851    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ab2f697-94f3-4b6a-928c-c6ef38b81c14" containerName="revoke-keys"
Oct 30 19:44:18.299970 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:18.299858    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="79d37674-5d73-4b92-a57f-66bee5d7574c" containerName="main"
Oct 30 19:44:18.304655 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:18.304634    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ac37c8-fqfbn"
Oct 30 19:44:18.312548 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:18.312518    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:44:18.312697 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:18.312564    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:44:18.312697 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:18.312645    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:44:18.312697 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:18.312647    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:44:18.318464 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:18.318437    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ac37c8-fqfbn"]
Oct 30 19:44:18.380371 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:18.380334    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e417b0f-a2b8-4792-9733-76b3a97a33df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ac37c8-fqfbn\" (UID: \"9e417b0f-a2b8-4792-9733-76b3a97a33df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ac37c8-fqfbn"
Oct 30 19:44:18.380800 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:18.380777    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e417b0f-a2b8-4792-9733-76b3a97a33df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ac37c8-fqfbn\" (UID: \"9e417b0f-a2b8-4792-9733-76b3a97a33df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ac37c8-fqfbn"
Oct 30 19:44:18.380939 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:18.380903    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qvkg\" (UniqueName: \"kubernetes.io/projected/9e417b0f-a2b8-4792-9733-76b3a97a33df-kube-api-access-6qvkg\") pod \"maas-api-revoke-keys-e2e-inf-b-ac37c8-fqfbn\" (UID: \"9e417b0f-a2b8-4792-9733-76b3a97a33df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ac37c8-fqfbn"
Oct 30 19:44:18.482174 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:18.482142    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qvkg\" (UniqueName: \"kubernetes.io/projected/9e417b0f-a2b8-4792-9733-76b3a97a33df-kube-api-access-6qvkg\") pod \"maas-api-revoke-keys-e2e-inf-b-ac37c8-fqfbn\" (UID: \"9e417b0f-a2b8-4792-9733-76b3a97a33df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ac37c8-fqfbn"
Oct 30 19:44:18.482364 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:18.482210    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e417b0f-a2b8-4792-9733-76b3a97a33df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ac37c8-fqfbn\" (UID: \"9e417b0f-a2b8-4792-9733-76b3a97a33df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ac37c8-fqfbn"
Oct 30 19:44:18.482364 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:18.482333    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e417b0f-a2b8-4792-9733-76b3a97a33df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ac37c8-fqfbn\" (UID: \"9e417b0f-a2b8-4792-9733-76b3a97a33df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ac37c8-fqfbn"
Oct 30 19:44:18.483098 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:18.483068    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e417b0f-a2b8-4792-9733-76b3a97a33df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ac37c8-fqfbn\" (UID: \"9e417b0f-a2b8-4792-9733-76b3a97a33df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ac37c8-fqfbn"
Oct 30 19:44:18.484967 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:18.484944    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e417b0f-a2b8-4792-9733-76b3a97a33df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ac37c8-fqfbn\" (UID: \"9e417b0f-a2b8-4792-9733-76b3a97a33df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ac37c8-fqfbn"
Oct 30 19:44:18.499565 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:18.499537    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qvkg\" (UniqueName: \"kubernetes.io/projected/9e417b0f-a2b8-4792-9733-76b3a97a33df-kube-api-access-6qvkg\") pod \"maas-api-revoke-keys-e2e-inf-b-ac37c8-fqfbn\" (UID: \"9e417b0f-a2b8-4792-9733-76b3a97a33df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ac37c8-fqfbn"
Oct 30 19:44:18.621529 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:18.621432    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ac37c8-fqfbn"
Oct 30 19:44:18.914916 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:18.914885    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ac37c8-fqfbn"]
Oct 30 19:44:18.915530 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:44:18.915509    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e417b0f_a2b8_4792_9733_76b3a97a33df.slice/crio-c4a931966a9110df096b8b5753a28b5988080eb75317736ac3c937de2e94ab06 WatchSource:0}: Error finding container c4a931966a9110df096b8b5753a28b5988080eb75317736ac3c937de2e94ab06: Status 404 returned error can't find the container with id c4a931966a9110df096b8b5753a28b5988080eb75317736ac3c937de2e94ab06
Oct 30 19:44:19.275756 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.275706    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89a3f6-6gmm5"]
Oct 30 19:44:19.282015 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.281983    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89a3f6-6gmm5"]
Oct 30 19:44:19.350806 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.350771    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ac37c8-fqfbn" event={"ID":"9e417b0f-a2b8-4792-9733-76b3a97a33df","Type":"ContainerStarted","Data":"c4a931966a9110df096b8b5753a28b5988080eb75317736ac3c937de2e94ab06"}
Oct 30 19:44:19.353093 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.353061    5043 generic.go:356] "Generic (PLEG): container finished" podID="c5c180ba-4a6f-4fb5-9976-1bd4b3526081" containerID="5479dc4f6f1bee401de9e545375dbdc8b111f8d453118eb9c4e84b4535fb8a02" exitCode=0
Oct 30 19:44:19.353247 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.353119    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf" event={"ID":"c5c180ba-4a6f-4fb5-9976-1bd4b3526081","Type":"ContainerDied","Data":"5479dc4f6f1bee401de9e545375dbdc8b111f8d453118eb9c4e84b4535fb8a02"}
Oct 30 19:44:19.355115 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.355091    5043 generic.go:356] "Generic (PLEG): container finished" podID="53142e30-10c0-413a-ae61-fb1b29f5fd10" containerID="a3b8b859589480824dd49f3184380eb6d291f261a3f0e93a16262fc4419e998f" exitCode=0
Oct 30 19:44:19.355224 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.355167    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc" event={"ID":"53142e30-10c0-413a-ae61-fb1b29f5fd10","Type":"ContainerDied","Data":"a3b8b859589480824dd49f3184380eb6d291f261a3f0e93a16262fc4419e998f"}
Oct 30 19:44:19.400692 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.400668    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf"
Oct 30 19:44:19.492713 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.492671    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89a3f6-db46ffd87-8l4s5" podUID="62c3de5f-9601-43f4-a89a-85c623e1222b" containerName="payload-pre-processing" containerID="cri-o://7ad16887948a82336cccb54ea5544800808823a9636386c02057bbaa8ac5316f" gracePeriod=25
Oct 30 19:44:19.509858 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.509833    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-credential-socket\") pod \"c5c180ba-4a6f-4fb5-9976-1bd4b3526081\" (UID: \"c5c180ba-4a6f-4fb5-9976-1bd4b3526081\") "
Oct 30 19:44:19.510007 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.509879    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-istio-token\" (UniqueName: \"kubernetes.io/projected/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-istio-token\") pod \"c5c180ba-4a6f-4fb5-9976-1bd4b3526081\" (UID: \"c5c180ba-4a6f-4fb5-9976-1bd4b3526081\") "
Oct 30 19:44:19.510007 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.509912    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-istio-data\") pod \"c5c180ba-4a6f-4fb5-9976-1bd4b3526081\" (UID: \"c5c180ba-4a6f-4fb5-9976-1bd4b3526081\") "
Oct 30 19:44:19.510007 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.509945    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-istio-envoy\") pod \"c5c180ba-4a6f-4fb5-9976-1bd4b3526081\" (UID: \"c5c180ba-4a6f-4fb5-9976-1bd4b3526081\") "
Oct 30 19:44:19.510193 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.510044    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-workload-socket\") pod \"c5c180ba-4a6f-4fb5-9976-1bd4b3526081\" (UID: \"c5c180ba-4a6f-4fb5-9976-1bd4b3526081\") "
Oct 30 19:44:19.510193 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.510092    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-istiod-ca-cert\") pod \"c5c180ba-4a6f-4fb5-9976-1bd4b3526081\" (UID: \"c5c180ba-4a6f-4fb5-9976-1bd4b3526081\") "
Oct 30 19:44:19.510193 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.510129    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-kube-api-access-l6sn9\" (UniqueName: \"kubernetes.io/projected/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-kube-api-access-l6sn9\") pod \"c5c180ba-4a6f-4fb5-9976-1bd4b3526081\" (UID: \"c5c180ba-4a6f-4fb5-9976-1bd4b3526081\") "
Oct 30 19:44:19.510193 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.510168    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-credential-socket" pod "c5c180ba-4a6f-4fb5-9976-1bd4b3526081" (UID: "c5c180ba-4a6f-4fb5-9976-1bd4b3526081"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:19.510411 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.510291    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-istio-podinfo\") pod \"c5c180ba-4a6f-4fb5-9976-1bd4b3526081\" (UID: \"c5c180ba-4a6f-4fb5-9976-1bd4b3526081\") "
Oct 30 19:44:19.510411 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.510337    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-istio-data" pod "c5c180ba-4a6f-4fb5-9976-1bd4b3526081" (UID: "c5c180ba-4a6f-4fb5-9976-1bd4b3526081"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:19.510411 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.510392    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-workload-socket" pod "c5c180ba-4a6f-4fb5-9976-1bd4b3526081" (UID: "c5c180ba-4a6f-4fb5-9976-1bd4b3526081"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:19.510550 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.510480    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-workload-certs\") pod \"c5c180ba-4a6f-4fb5-9976-1bd4b3526081\" (UID: \"c5c180ba-4a6f-4fb5-9976-1bd4b3526081\") "
Oct 30 19:44:19.510612 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.510581    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-istiod-ca-cert" pod "c5c180ba-4a6f-4fb5-9976-1bd4b3526081" (UID: "c5c180ba-4a6f-4fb5-9976-1bd4b3526081"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:19.510776 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.510726    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-workload-certs" pod "c5c180ba-4a6f-4fb5-9976-1bd4b3526081" (UID: "c5c180ba-4a6f-4fb5-9976-1bd4b3526081"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:19.511414 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.511392    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:19.511479 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.511424    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:19.511479 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.511440    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:19.511479 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.511456    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:19.511479 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.511470    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:19.512331 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.512313    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-istio-envoy" pod "c5c180ba-4a6f-4fb5-9976-1bd4b3526081" (UID: "c5c180ba-4a6f-4fb5-9976-1bd4b3526081"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:19.512849 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.512820    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-istio-podinfo" pod "c5c180ba-4a6f-4fb5-9976-1bd4b3526081" (UID: "c5c180ba-4a6f-4fb5-9976-1bd4b3526081"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:44:19.512936 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.512882    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-istio-token" pod "c5c180ba-4a6f-4fb5-9976-1bd4b3526081" (UID: "c5c180ba-4a6f-4fb5-9976-1bd4b3526081"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:19.512985 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.512930    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-kube-api-access-l6sn9" pod "c5c180ba-4a6f-4fb5-9976-1bd4b3526081" (UID: "c5c180ba-4a6f-4fb5-9976-1bd4b3526081"). InnerVolumeSpecName "kube-api-access-l6sn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:19.613613 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.613571    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:19.613613 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.613617    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:19.614038 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.613634    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6sn9\" (UniqueName: \"kubernetes.io/projected/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-kube-api-access-l6sn9\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:19.614038 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.613651    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5c180ba-4a6f-4fb5-9976-1bd4b3526081-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:19.671415 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.671374    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89a3f6-db46ffd87-8l4s5" podUID="62c3de5f-9601-43f4-a89a-85c623e1222b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:44:19.776141 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.776113    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc"
Oct 30 19:44:19.824982 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.824944    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53142e30-10c0-413a-ae61-fb1b29f5fd10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53142e30-10c0-413a-ae61-fb1b29f5fd10-istio-podinfo\") pod \"53142e30-10c0-413a-ae61-fb1b29f5fd10\" (UID: \"53142e30-10c0-413a-ae61-fb1b29f5fd10\") "
Oct 30 19:44:19.825192 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.825046    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53142e30-10c0-413a-ae61-fb1b29f5fd10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53142e30-10c0-413a-ae61-fb1b29f5fd10-istio-data\") pod \"53142e30-10c0-413a-ae61-fb1b29f5fd10\" (UID: \"53142e30-10c0-413a-ae61-fb1b29f5fd10\") "
Oct 30 19:44:19.825192 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.825129    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53142e30-10c0-413a-ae61-fb1b29f5fd10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53142e30-10c0-413a-ae61-fb1b29f5fd10-credential-socket\") pod \"53142e30-10c0-413a-ae61-fb1b29f5fd10\" (UID: \"53142e30-10c0-413a-ae61-fb1b29f5fd10\") "
Oct 30 19:44:19.825192 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.825190    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53142e30-10c0-413a-ae61-fb1b29f5fd10-istio-token\" (UniqueName: \"kubernetes.io/projected/53142e30-10c0-413a-ae61-fb1b29f5fd10-istio-token\") pod \"53142e30-10c0-413a-ae61-fb1b29f5fd10\" (UID: \"53142e30-10c0-413a-ae61-fb1b29f5fd10\") "
Oct 30 19:44:19.825382 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.825229    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53142e30-10c0-413a-ae61-fb1b29f5fd10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53142e30-10c0-413a-ae61-fb1b29f5fd10-istio-envoy\") pod \"53142e30-10c0-413a-ae61-fb1b29f5fd10\" (UID: \"53142e30-10c0-413a-ae61-fb1b29f5fd10\") "
Oct 30 19:44:19.825382 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.825265    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53142e30-10c0-413a-ae61-fb1b29f5fd10-kube-api-access-54fp6\" (UniqueName: \"kubernetes.io/projected/53142e30-10c0-413a-ae61-fb1b29f5fd10-kube-api-access-54fp6\") pod \"53142e30-10c0-413a-ae61-fb1b29f5fd10\" (UID: \"53142e30-10c0-413a-ae61-fb1b29f5fd10\") "
Oct 30 19:44:19.825382 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.825298    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53142e30-10c0-413a-ae61-fb1b29f5fd10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53142e30-10c0-413a-ae61-fb1b29f5fd10-istiod-ca-cert\") pod \"53142e30-10c0-413a-ae61-fb1b29f5fd10\" (UID: \"53142e30-10c0-413a-ae61-fb1b29f5fd10\") "
Oct 30 19:44:19.825382 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.825315    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53142e30-10c0-413a-ae61-fb1b29f5fd10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53142e30-10c0-413a-ae61-fb1b29f5fd10-workload-socket\") pod \"53142e30-10c0-413a-ae61-fb1b29f5fd10\" (UID: \"53142e30-10c0-413a-ae61-fb1b29f5fd10\") "
Oct 30 19:44:19.825382 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.825345    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53142e30-10c0-413a-ae61-fb1b29f5fd10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53142e30-10c0-413a-ae61-fb1b29f5fd10-workload-certs\") pod \"53142e30-10c0-413a-ae61-fb1b29f5fd10\" (UID: \"53142e30-10c0-413a-ae61-fb1b29f5fd10\") "
Oct 30 19:44:19.826500 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.826275    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53142e30-10c0-413a-ae61-fb1b29f5fd10-istio-data" pod "53142e30-10c0-413a-ae61-fb1b29f5fd10" (UID: "53142e30-10c0-413a-ae61-fb1b29f5fd10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:19.826500 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.826297    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53142e30-10c0-413a-ae61-fb1b29f5fd10-workload-certs" pod "53142e30-10c0-413a-ae61-fb1b29f5fd10" (UID: "53142e30-10c0-413a-ae61-fb1b29f5fd10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:19.826500 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.826373    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53142e30-10c0-413a-ae61-fb1b29f5fd10-workload-socket" pod "53142e30-10c0-413a-ae61-fb1b29f5fd10" (UID: "53142e30-10c0-413a-ae61-fb1b29f5fd10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:19.827425 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.827356    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53142e30-10c0-413a-ae61-fb1b29f5fd10-credential-socket" pod "53142e30-10c0-413a-ae61-fb1b29f5fd10" (UID: "53142e30-10c0-413a-ae61-fb1b29f5fd10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:19.837268 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.837135    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53142e30-10c0-413a-ae61-fb1b29f5fd10-istio-token" pod "53142e30-10c0-413a-ae61-fb1b29f5fd10" (UID: "53142e30-10c0-413a-ae61-fb1b29f5fd10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:19.860758 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.849993    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53142e30-10c0-413a-ae61-fb1b29f5fd10-istiod-ca-cert" pod "53142e30-10c0-413a-ae61-fb1b29f5fd10" (UID: "53142e30-10c0-413a-ae61-fb1b29f5fd10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:19.860758 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.855390    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53142e30-10c0-413a-ae61-fb1b29f5fd10-kube-api-access-54fp6" pod "53142e30-10c0-413a-ae61-fb1b29f5fd10" (UID: "53142e30-10c0-413a-ae61-fb1b29f5fd10"). InnerVolumeSpecName "kube-api-access-54fp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:19.875413 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.874872    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53142e30-10c0-413a-ae61-fb1b29f5fd10-istio-podinfo" pod "53142e30-10c0-413a-ae61-fb1b29f5fd10" (UID: "53142e30-10c0-413a-ae61-fb1b29f5fd10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:44:19.916214 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.916168    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53142e30-10c0-413a-ae61-fb1b29f5fd10-istio-envoy" pod "53142e30-10c0-413a-ae61-fb1b29f5fd10" (UID: "53142e30-10c0-413a-ae61-fb1b29f5fd10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:19.945695 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.938401    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53142e30-10c0-413a-ae61-fb1b29f5fd10-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:19.945695 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.938438    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53142e30-10c0-413a-ae61-fb1b29f5fd10-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:19.945695 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.938455    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53142e30-10c0-413a-ae61-fb1b29f5fd10-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:19.945695 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.938471    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54fp6\" (UniqueName: \"kubernetes.io/projected/53142e30-10c0-413a-ae61-fb1b29f5fd10-kube-api-access-54fp6\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:19.945695 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.938487    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53142e30-10c0-413a-ae61-fb1b29f5fd10-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:19.945695 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.938499    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53142e30-10c0-413a-ae61-fb1b29f5fd10-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:19.945695 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.938512    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53142e30-10c0-413a-ae61-fb1b29f5fd10-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:19.945695 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.938525    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53142e30-10c0-413a-ae61-fb1b29f5fd10-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:19.945695 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:19.938538    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53142e30-10c0-413a-ae61-fb1b29f5fd10-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:20.364644 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:20.364574    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf"
Oct 30 19:44:20.364942 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:20.364573    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf" event={"ID":"c5c180ba-4a6f-4fb5-9976-1bd4b3526081","Type":"ContainerDied","Data":"2ace47431e507256cda5e53f090b15021dd58d0f02f816db0c9e6572912123c5"}
Oct 30 19:44:20.364942 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:20.364718    5043 scope.go:122] "RemoveContainer" containerID="5479dc4f6f1bee401de9e545375dbdc8b111f8d453118eb9c4e84b4535fb8a02"
Oct 30 19:44:20.366253 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:20.366236    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc"
Oct 30 19:44:20.366343 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:20.366323    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc" event={"ID":"53142e30-10c0-413a-ae61-fb1b29f5fd10","Type":"ContainerDied","Data":"c09e32219273785bdbd5d58e66490410ee74b904defaba950163b3128a9c7e31"}
Oct 30 19:44:20.368216 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:20.368190    5043 generic.go:356] "Generic (PLEG): container finished" podID="9e417b0f-a2b8-4792-9733-76b3a97a33df" containerID="dae1ddb6be0aec4fc67a7f9f6a1a4e1fbc72c0308f6bb8dadcd881569e02a3b1" exitCode=0
Oct 30 19:44:20.368346 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:20.368299    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ac37c8-fqfbn" event={"ID":"9e417b0f-a2b8-4792-9733-76b3a97a33df","Type":"ContainerDied","Data":"dae1ddb6be0aec4fc67a7f9f6a1a4e1fbc72c0308f6bb8dadcd881569e02a3b1"}
Oct 30 19:44:20.378690 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:20.378674    5043 scope.go:122] "RemoveContainer" containerID="a3b8b859589480824dd49f3184380eb6d291f261a3f0e93a16262fc4419e998f"
Oct 30 19:44:20.423245 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:20.423201    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc"]
Oct 30 19:44:20.431516 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:20.431485    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-fpzgc"]
Oct 30 19:44:20.448843 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:20.448814    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf"]
Oct 30 19:44:20.461606 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:20.461575    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-gqmnf"]
Oct 30 19:44:20.647347 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:20.647269    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ab2f697-94f3-4b6a-928c-c6ef38b81c14" path="/var/lib/kubelet/pods/1ab2f697-94f3-4b6a-928c-c6ef38b81c14/volumes"
Oct 30 19:44:20.647875 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:20.647855    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53142e30-10c0-413a-ae61-fb1b29f5fd10" path="/var/lib/kubelet/pods/53142e30-10c0-413a-ae61-fb1b29f5fd10/volumes"
Oct 30 19:44:20.648481 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:20.648465    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5c180ba-4a6f-4fb5-9976-1bd4b3526081" path="/var/lib/kubelet/pods/c5c180ba-4a6f-4fb5-9976-1bd4b3526081/volumes"
Oct 30 19:44:21.572654 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:21.572628    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ac37c8-fqfbn"
Oct 30 19:44:21.660526 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:21.660439    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e417b0f-a2b8-4792-9733-76b3a97a33df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e417b0f-a2b8-4792-9733-76b3a97a33df-maas-api-openshift-ca\") pod \"9e417b0f-a2b8-4792-9733-76b3a97a33df\" (UID: \"9e417b0f-a2b8-4792-9733-76b3a97a33df\") "
Oct 30 19:44:21.660699 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:21.660533    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e417b0f-a2b8-4792-9733-76b3a97a33df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e417b0f-a2b8-4792-9733-76b3a97a33df-maas-api-xks-ca\") pod \"9e417b0f-a2b8-4792-9733-76b3a97a33df\" (UID: \"9e417b0f-a2b8-4792-9733-76b3a97a33df\") "
Oct 30 19:44:21.660699 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:21.660596    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e417b0f-a2b8-4792-9733-76b3a97a33df-kube-api-access-6qvkg\" (UniqueName: \"kubernetes.io/projected/9e417b0f-a2b8-4792-9733-76b3a97a33df-kube-api-access-6qvkg\") pod \"9e417b0f-a2b8-4792-9733-76b3a97a33df\" (UID: \"9e417b0f-a2b8-4792-9733-76b3a97a33df\") "
Oct 30 19:44:21.662012 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:21.661921    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e417b0f-a2b8-4792-9733-76b3a97a33df-maas-api-openshift-ca" pod "9e417b0f-a2b8-4792-9733-76b3a97a33df" (UID: "9e417b0f-a2b8-4792-9733-76b3a97a33df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:21.680186 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:21.680142    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e417b0f-a2b8-4792-9733-76b3a97a33df-kube-api-access-6qvkg" pod "9e417b0f-a2b8-4792-9733-76b3a97a33df" (UID: "9e417b0f-a2b8-4792-9733-76b3a97a33df"). InnerVolumeSpecName "kube-api-access-6qvkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:21.681611 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:21.681575    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e417b0f-a2b8-4792-9733-76b3a97a33df-maas-api-xks-ca" pod "9e417b0f-a2b8-4792-9733-76b3a97a33df" (UID: "9e417b0f-a2b8-4792-9733-76b3a97a33df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:44:21.762300 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:21.761937    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96"
Oct 30 19:44:21.763253 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:21.763225    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e417b0f-a2b8-4792-9733-76b3a97a33df-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:21.763415 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:21.763259    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qvkg\" (UniqueName: \"kubernetes.io/projected/9e417b0f-a2b8-4792-9733-76b3a97a33df-kube-api-access-6qvkg\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:21.763415 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:21.763277    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e417b0f-a2b8-4792-9733-76b3a97a33df-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:21.864471 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:21.864430    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/377f3c78-650d-4680-aa0f-0373eba791eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/377f3c78-650d-4680-aa0f-0373eba791eb-istio-podinfo\") pod \"377f3c78-650d-4680-aa0f-0373eba791eb\" (UID: \"377f3c78-650d-4680-aa0f-0373eba791eb\") "
Oct 30 19:44:21.864667 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:21.864485    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/377f3c78-650d-4680-aa0f-0373eba791eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/377f3c78-650d-4680-aa0f-0373eba791eb-istio-envoy\") pod \"377f3c78-650d-4680-aa0f-0373eba791eb\" (UID: \"377f3c78-650d-4680-aa0f-0373eba791eb\") "
Oct 30 19:44:21.864667 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:21.864556    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/377f3c78-650d-4680-aa0f-0373eba791eb-kube-api-access-cw87q\" (UniqueName: \"kubernetes.io/projected/377f3c78-650d-4680-aa0f-0373eba791eb-kube-api-access-cw87q\") pod \"377f3c78-650d-4680-aa0f-0373eba791eb\" (UID: \"377f3c78-650d-4680-aa0f-0373eba791eb\") "
Oct 30 19:44:21.864667 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:21.864599    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/377f3c78-650d-4680-aa0f-0373eba791eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/377f3c78-650d-4680-aa0f-0373eba791eb-workload-certs\") pod \"377f3c78-650d-4680-aa0f-0373eba791eb\" (UID: \"377f3c78-650d-4680-aa0f-0373eba791eb\") "
Oct 30 19:44:21.864667 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:21.864642    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/377f3c78-650d-4680-aa0f-0373eba791eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/377f3c78-650d-4680-aa0f-0373eba791eb-credential-socket\") pod \"377f3c78-650d-4680-aa0f-0373eba791eb\" (UID: \"377f3c78-650d-4680-aa0f-0373eba791eb\") "
Oct 30 19:44:21.864961 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:21.864727    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/377f3c78-650d-4680-aa0f-0373eba791eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/377f3c78-650d-4680-aa0f-0373eba791eb-istiod-ca-cert\") pod \"377f3c78-650d-4680-aa0f-0373eba791eb\" (UID: \"377f3c78-650d-4680-aa0f-0373eba791eb\") "
Oct 30 19:44:21.864961 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:21.864878    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/377f3c78-650d-4680-aa0f-0373eba791eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/377f3c78-650d-4680-aa0f-0373eba791eb-workload-socket\") pod \"377f3c78-650d-4680-aa0f-0373eba791eb\" (UID: \"377f3c78-650d-4680-aa0f-0373eba791eb\") "
Oct 30 19:44:21.865073 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:21.864989    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377f3c78-650d-4680-aa0f-0373eba791eb-workload-certs" pod "377f3c78-650d-4680-aa0f-0373eba791eb" (UID: "377f3c78-650d-4680-aa0f-0373eba791eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:21.865073 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:21.865002    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/377f3c78-650d-4680-aa0f-0373eba791eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/377f3c78-650d-4680-aa0f-0373eba791eb-istio-data\") pod \"377f3c78-650d-4680-aa0f-0373eba791eb\" (UID: \"377f3c78-650d-4680-aa0f-0373eba791eb\") "
Oct 30 19:44:21.865194 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:21.865099    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/377f3c78-650d-4680-aa0f-0373eba791eb-istio-token\" (UniqueName: \"kubernetes.io/projected/377f3c78-650d-4680-aa0f-0373eba791eb-istio-token\") pod \"377f3c78-650d-4680-aa0f-0373eba791eb\" (UID: \"377f3c78-650d-4680-aa0f-0373eba791eb\") "
Oct 30 19:44:21.865194 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:21.865182    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377f3c78-650d-4680-aa0f-0373eba791eb-istio-data" pod "377f3c78-650d-4680-aa0f-0373eba791eb" (UID: "377f3c78-650d-4680-aa0f-0373eba791eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:21.865337 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:21.865314    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377f3c78-650d-4680-aa0f-0373eba791eb-workload-socket" pod "377f3c78-650d-4680-aa0f-0373eba791eb" (UID: "377f3c78-650d-4680-aa0f-0373eba791eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:21.865396 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:21.865380    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377f3c78-650d-4680-aa0f-0373eba791eb-credential-socket" pod "377f3c78-650d-4680-aa0f-0373eba791eb" (UID: "377f3c78-650d-4680-aa0f-0373eba791eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:21.865537 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:21.865520    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/377f3c78-650d-4680-aa0f-0373eba791eb-istiod-ca-cert" pod "377f3c78-650d-4680-aa0f-0373eba791eb" (UID: "377f3c78-650d-4680-aa0f-0373eba791eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:21.865956 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:21.865892    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/377f3c78-650d-4680-aa0f-0373eba791eb-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:21.866812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:21.866038    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/377f3c78-650d-4680-aa0f-0373eba791eb-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:21.866812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:21.866059    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/377f3c78-650d-4680-aa0f-0373eba791eb-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:21.866812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:21.866074    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/377f3c78-650d-4680-aa0f-0373eba791eb-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:21.866812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:21.866089    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/377f3c78-650d-4680-aa0f-0373eba791eb-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:21.867176 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:21.867151    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377f3c78-650d-4680-aa0f-0373eba791eb-istio-envoy" pod "377f3c78-650d-4680-aa0f-0373eba791eb" (UID: "377f3c78-650d-4680-aa0f-0373eba791eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:21.867320 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:21.867303    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/377f3c78-650d-4680-aa0f-0373eba791eb-kube-api-access-cw87q" pod "377f3c78-650d-4680-aa0f-0373eba791eb" (UID: "377f3c78-650d-4680-aa0f-0373eba791eb"). InnerVolumeSpecName "kube-api-access-cw87q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:21.867517 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:21.867499    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/377f3c78-650d-4680-aa0f-0373eba791eb-istio-podinfo" pod "377f3c78-650d-4680-aa0f-0373eba791eb" (UID: "377f3c78-650d-4680-aa0f-0373eba791eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:44:21.867897 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:21.867881    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/377f3c78-650d-4680-aa0f-0373eba791eb-istio-token" pod "377f3c78-650d-4680-aa0f-0373eba791eb" (UID: "377f3c78-650d-4680-aa0f-0373eba791eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:21.967681 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:21.967650    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/377f3c78-650d-4680-aa0f-0373eba791eb-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:21.967681 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:21.967682    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/377f3c78-650d-4680-aa0f-0373eba791eb-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:21.967927 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:21.967692    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/377f3c78-650d-4680-aa0f-0373eba791eb-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:21.967927 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:21.967702    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw87q\" (UniqueName: \"kubernetes.io/projected/377f3c78-650d-4680-aa0f-0373eba791eb-kube-api-access-cw87q\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:22.388073 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:22.387985    5043 generic.go:356] "Generic (PLEG): container finished" podID="377f3c78-650d-4680-aa0f-0373eba791eb" containerID="7f2abe2aab6157719288b52cb9c9cf58ac9724e3b060f8c88bf92e623d1784fa" exitCode=0
Oct 30 19:44:22.388073 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:22.388032    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96"
Oct 30 19:44:22.388073 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:22.388061    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96" event={"ID":"377f3c78-650d-4680-aa0f-0373eba791eb","Type":"ContainerDied","Data":"7f2abe2aab6157719288b52cb9c9cf58ac9724e3b060f8c88bf92e623d1784fa"}
Oct 30 19:44:22.388366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:22.388121    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96" event={"ID":"377f3c78-650d-4680-aa0f-0373eba791eb","Type":"ContainerDied","Data":"5df2da6cb055d8b4d01256b3f5cd7cb74229839f3fe6921534417630dd8983f3"}
Oct 30 19:44:22.388366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:22.388151    5043 scope.go:122] "RemoveContainer" containerID="7f2abe2aab6157719288b52cb9c9cf58ac9724e3b060f8c88bf92e623d1784fa"
Oct 30 19:44:22.391779 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:22.389866    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ac37c8-fqfbn" event={"ID":"9e417b0f-a2b8-4792-9733-76b3a97a33df","Type":"ContainerDied","Data":"c4a931966a9110df096b8b5753a28b5988080eb75317736ac3c937de2e94ab06"}
Oct 30 19:44:22.391779 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:22.389909    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4a931966a9110df096b8b5753a28b5988080eb75317736ac3c937de2e94ab06"
Oct 30 19:44:22.391779 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:22.389873    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ac37c8-fqfbn"
Oct 30 19:44:22.400790 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:22.400771    5043 scope.go:122] "RemoveContainer" containerID="7f2abe2aab6157719288b52cb9c9cf58ac9724e3b060f8c88bf92e623d1784fa"
Oct 30 19:44:22.401073 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:44:22.401055    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f2abe2aab6157719288b52cb9c9cf58ac9724e3b060f8c88bf92e623d1784fa\": container with ID starting with 7f2abe2aab6157719288b52cb9c9cf58ac9724e3b060f8c88bf92e623d1784fa not found: ID does not exist" containerID="7f2abe2aab6157719288b52cb9c9cf58ac9724e3b060f8c88bf92e623d1784fa"
Oct 30 19:44:22.401121 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:22.401081    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f2abe2aab6157719288b52cb9c9cf58ac9724e3b060f8c88bf92e623d1784fa"} err="failed to get container status \"7f2abe2aab6157719288b52cb9c9cf58ac9724e3b060f8c88bf92e623d1784fa\": rpc error: code = NotFound desc = could not find container \"7f2abe2aab6157719288b52cb9c9cf58ac9724e3b060f8c88bf92e623d1784fa\": container with ID starting with 7f2abe2aab6157719288b52cb9c9cf58ac9724e3b060f8c88bf92e623d1784fa not found: ID does not exist"
Oct 30 19:44:22.424121 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:22.424086    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96"]
Oct 30 19:44:22.427654 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:22.427626    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qdh96"]
Oct 30 19:44:22.649994 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:22.649909    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="377f3c78-650d-4680-aa0f-0373eba791eb" path="/var/lib/kubelet/pods/377f3c78-650d-4680-aa0f-0373eba791eb/volumes"
Oct 30 19:44:23.121294 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:23.121265    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd"
Oct 30 19:44:23.183369 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:23.183280    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ff1923b-6547-4e9b-8810-3f79648e3272-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ff1923b-6547-4e9b-8810-3f79648e3272-istio-podinfo\") pod \"5ff1923b-6547-4e9b-8810-3f79648e3272\" (UID: \"5ff1923b-6547-4e9b-8810-3f79648e3272\") "
Oct 30 19:44:23.183369 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:23.183333    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ff1923b-6547-4e9b-8810-3f79648e3272-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ff1923b-6547-4e9b-8810-3f79648e3272-credential-socket\") pod \"5ff1923b-6547-4e9b-8810-3f79648e3272\" (UID: \"5ff1923b-6547-4e9b-8810-3f79648e3272\") "
Oct 30 19:44:23.183369 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:23.183357    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ff1923b-6547-4e9b-8810-3f79648e3272-kube-api-access-qd494\" (UniqueName: \"kubernetes.io/projected/5ff1923b-6547-4e9b-8810-3f79648e3272-kube-api-access-qd494\") pod \"5ff1923b-6547-4e9b-8810-3f79648e3272\" (UID: \"5ff1923b-6547-4e9b-8810-3f79648e3272\") "
Oct 30 19:44:23.183676 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:23.183444    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ff1923b-6547-4e9b-8810-3f79648e3272-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ff1923b-6547-4e9b-8810-3f79648e3272-istiod-ca-cert\") pod \"5ff1923b-6547-4e9b-8810-3f79648e3272\" (UID: \"5ff1923b-6547-4e9b-8810-3f79648e3272\") "
Oct 30 19:44:23.183676 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:23.183474    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ff1923b-6547-4e9b-8810-3f79648e3272-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ff1923b-6547-4e9b-8810-3f79648e3272-istio-envoy\") pod \"5ff1923b-6547-4e9b-8810-3f79648e3272\" (UID: \"5ff1923b-6547-4e9b-8810-3f79648e3272\") "
Oct 30 19:44:23.183676 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:23.183551    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ff1923b-6547-4e9b-8810-3f79648e3272-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ff1923b-6547-4e9b-8810-3f79648e3272-workload-socket\") pod \"5ff1923b-6547-4e9b-8810-3f79648e3272\" (UID: \"5ff1923b-6547-4e9b-8810-3f79648e3272\") "
Oct 30 19:44:23.183676 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:23.183608    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ff1923b-6547-4e9b-8810-3f79648e3272-istio-token\" (UniqueName: \"kubernetes.io/projected/5ff1923b-6547-4e9b-8810-3f79648e3272-istio-token\") pod \"5ff1923b-6547-4e9b-8810-3f79648e3272\" (UID: \"5ff1923b-6547-4e9b-8810-3f79648e3272\") "
Oct 30 19:44:23.183676 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:23.183636    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ff1923b-6547-4e9b-8810-3f79648e3272-credential-socket" pod "5ff1923b-6547-4e9b-8810-3f79648e3272" (UID: "5ff1923b-6547-4e9b-8810-3f79648e3272"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:23.183676 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:23.183659    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ff1923b-6547-4e9b-8810-3f79648e3272-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ff1923b-6547-4e9b-8810-3f79648e3272-workload-certs\") pod \"5ff1923b-6547-4e9b-8810-3f79648e3272\" (UID: \"5ff1923b-6547-4e9b-8810-3f79648e3272\") "
Oct 30 19:44:23.184124 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:23.183810    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ff1923b-6547-4e9b-8810-3f79648e3272-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ff1923b-6547-4e9b-8810-3f79648e3272-istio-data\") pod \"5ff1923b-6547-4e9b-8810-3f79648e3272\" (UID: \"5ff1923b-6547-4e9b-8810-3f79648e3272\") "
Oct 30 19:44:23.184223 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:23.184205    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ff1923b-6547-4e9b-8810-3f79648e3272-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:23.184384 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:23.184366    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ff1923b-6547-4e9b-8810-3f79648e3272-istio-data" pod "5ff1923b-6547-4e9b-8810-3f79648e3272" (UID: "5ff1923b-6547-4e9b-8810-3f79648e3272"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:23.184629 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:23.184604    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ff1923b-6547-4e9b-8810-3f79648e3272-workload-certs" pod "5ff1923b-6547-4e9b-8810-3f79648e3272" (UID: "5ff1923b-6547-4e9b-8810-3f79648e3272"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:23.184770 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:23.184625    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ff1923b-6547-4e9b-8810-3f79648e3272-istiod-ca-cert" pod "5ff1923b-6547-4e9b-8810-3f79648e3272" (UID: "5ff1923b-6547-4e9b-8810-3f79648e3272"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:23.186152 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:23.186121    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ff1923b-6547-4e9b-8810-3f79648e3272-workload-socket" pod "5ff1923b-6547-4e9b-8810-3f79648e3272" (UID: "5ff1923b-6547-4e9b-8810-3f79648e3272"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:23.186949 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:23.186919    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ff1923b-6547-4e9b-8810-3f79648e3272-kube-api-access-qd494" pod "5ff1923b-6547-4e9b-8810-3f79648e3272" (UID: "5ff1923b-6547-4e9b-8810-3f79648e3272"). InnerVolumeSpecName "kube-api-access-qd494". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:23.188455 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:23.188431    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ff1923b-6547-4e9b-8810-3f79648e3272-istio-envoy" pod "5ff1923b-6547-4e9b-8810-3f79648e3272" (UID: "5ff1923b-6547-4e9b-8810-3f79648e3272"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:23.188983 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:23.188963    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ff1923b-6547-4e9b-8810-3f79648e3272-istio-podinfo" pod "5ff1923b-6547-4e9b-8810-3f79648e3272" (UID: "5ff1923b-6547-4e9b-8810-3f79648e3272"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:44:23.189430 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:23.189392    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ff1923b-6547-4e9b-8810-3f79648e3272-istio-token" pod "5ff1923b-6547-4e9b-8810-3f79648e3272" (UID: "5ff1923b-6547-4e9b-8810-3f79648e3272"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:23.284919 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:23.284886    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ff1923b-6547-4e9b-8810-3f79648e3272-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:23.284919 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:23.284912    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ff1923b-6547-4e9b-8810-3f79648e3272-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:23.284919 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:23.284923    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ff1923b-6547-4e9b-8810-3f79648e3272-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:23.285217 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:23.284933    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ff1923b-6547-4e9b-8810-3f79648e3272-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:23.285217 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:23.284942    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd494\" (UniqueName: \"kubernetes.io/projected/5ff1923b-6547-4e9b-8810-3f79648e3272-kube-api-access-qd494\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:23.285217 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:23.284951    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ff1923b-6547-4e9b-8810-3f79648e3272-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:23.285217 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:23.284959    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ff1923b-6547-4e9b-8810-3f79648e3272-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:23.285217 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:23.284968    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ff1923b-6547-4e9b-8810-3f79648e3272-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:23.420533 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:23.420495    5043 generic.go:356] "Generic (PLEG): container finished" podID="5ff1923b-6547-4e9b-8810-3f79648e3272" containerID="d4061dda458306dde91ff8f2a254ce9bb14709f6ea844c758630cff33d1aa8a4" exitCode=0
Oct 30 19:44:23.420802 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:23.420708    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd" event={"ID":"5ff1923b-6547-4e9b-8810-3f79648e3272","Type":"ContainerDied","Data":"d4061dda458306dde91ff8f2a254ce9bb14709f6ea844c758630cff33d1aa8a4"}
Oct 30 19:44:23.420894 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:23.420824    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd" event={"ID":"5ff1923b-6547-4e9b-8810-3f79648e3272","Type":"ContainerDied","Data":"0ee9c1430b6ccd0b2c31af897216adc76d3932093a50a28f3c5d83a0fbbb01a4"}
Oct 30 19:44:23.420894 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:23.420864    5043 scope.go:122] "RemoveContainer" containerID="d4061dda458306dde91ff8f2a254ce9bb14709f6ea844c758630cff33d1aa8a4"
Oct 30 19:44:23.420894 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:23.420865    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd"
Oct 30 19:44:23.445654 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:23.445636    5043 scope.go:122] "RemoveContainer" containerID="7ec243dad6c669aa3939aa3a8751708fa5f41444c4da0a374873dc8b9e1e3f16"
Oct 30 19:44:23.453928 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:23.453901    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd"]
Oct 30 19:44:23.459111 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:23.459084    5043 scope.go:122] "RemoveContainer" containerID="d4061dda458306dde91ff8f2a254ce9bb14709f6ea844c758630cff33d1aa8a4"
Oct 30 19:44:23.459500 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:44:23.459457    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4061dda458306dde91ff8f2a254ce9bb14709f6ea844c758630cff33d1aa8a4\": container with ID starting with d4061dda458306dde91ff8f2a254ce9bb14709f6ea844c758630cff33d1aa8a4 not found: ID does not exist" containerID="d4061dda458306dde91ff8f2a254ce9bb14709f6ea844c758630cff33d1aa8a4"
Oct 30 19:44:23.459604 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:23.459504    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4061dda458306dde91ff8f2a254ce9bb14709f6ea844c758630cff33d1aa8a4"} err="failed to get container status \"d4061dda458306dde91ff8f2a254ce9bb14709f6ea844c758630cff33d1aa8a4\": rpc error: code = NotFound desc = could not find container \"d4061dda458306dde91ff8f2a254ce9bb14709f6ea844c758630cff33d1aa8a4\": container with ID starting with d4061dda458306dde91ff8f2a254ce9bb14709f6ea844c758630cff33d1aa8a4 not found: ID does not exist"
Oct 30 19:44:23.459604 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:23.459524    5043 scope.go:122] "RemoveContainer" containerID="7ec243dad6c669aa3939aa3a8751708fa5f41444c4da0a374873dc8b9e1e3f16"
Oct 30 19:44:23.459927 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:44:23.459887    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ec243dad6c669aa3939aa3a8751708fa5f41444c4da0a374873dc8b9e1e3f16\": container with ID starting with 7ec243dad6c669aa3939aa3a8751708fa5f41444c4da0a374873dc8b9e1e3f16 not found: ID does not exist" containerID="7ec243dad6c669aa3939aa3a8751708fa5f41444c4da0a374873dc8b9e1e3f16"
Oct 30 19:44:23.460036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:23.459925    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ec243dad6c669aa3939aa3a8751708fa5f41444c4da0a374873dc8b9e1e3f16"} err="failed to get container status \"7ec243dad6c669aa3939aa3a8751708fa5f41444c4da0a374873dc8b9e1e3f16\": rpc error: code = NotFound desc = could not find container \"7ec243dad6c669aa3939aa3a8751708fa5f41444c4da0a374873dc8b9e1e3f16\": container with ID starting with 7ec243dad6c669aa3939aa3a8751708fa5f41444c4da0a374873dc8b9e1e3f16 not found: ID does not exist"
Oct 30 19:44:23.463295 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:23.463271    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-744b57-openshift-default-6c5cbf6bb9-zctmd"]
Oct 30 19:44:23.910781 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:23.910745    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-744b57-77cc66bfc9-2fcz9" podUID="4ef41111-49f3-4fd9-9587-2a1934f50797" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:44:24.527325 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:24.527284    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1e13df-78b7ccf4bc-8ldn2" podUID="3cd1e9e6-6a23-40d8-aa51-0266d9c61446" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:44:24.648541 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:24.648491    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ff1923b-6547-4e9b-8810-3f79648e3272" path="/var/lib/kubelet/pods/5ff1923b-6547-4e9b-8810-3f79648e3272/volumes"
Oct 30 19:44:24.660090 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:24.660063    5043 scope.go:122] "RemoveContainer" containerID="a5f721e18202d159d86aa018d44573d4d42069066d22449b917f5087fae5ebf7"
Oct 30 19:44:24.676781 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:24.676758    5043 scope.go:122] "RemoveContainer" containerID="c51b6a9d96b1f22f01da463d572792872c36f3b81a96cc893d6e22c43b955085"
Oct 30 19:44:24.686559 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:24.686541    5043 scope.go:122] "RemoveContainer" containerID="3ee0767c72ba5ca12fd92f11e4faf087f217ae19409366835f6b277951c2fabf"
Oct 30 19:44:24.696276 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:24.696251    5043 scope.go:122] "RemoveContainer" containerID="95ca287b5e5a4666f51105382ef510973597c87b3d018c47352ad5901dd830d3"
Oct 30 19:44:24.706084 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:24.706055    5043 scope.go:122] "RemoveContainer" containerID="f9ce5733f90ab37dd6011846ca9c8dd59626e8d4887ad81739d2db16f7140570"
Oct 30 19:44:24.978728 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:44:24.978686    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:44:25.496877 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:25.496847    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx"]
Oct 30 19:44:25.497109 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:25.497084    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx" podUID="f791cec1-a1e4-4067-9221-cc04dafba0eb" containerName="istio-proxy" containerID="cri-o://cb07df21823bf0822cc28be90ffa63cfdc67e892be4a904cc079f6e990725195" gracePeriod=30
Oct 30 19:44:26.261898 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:26.261861    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-01b24e-9gkpt"]
Oct 30 19:44:26.263105 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:26.263069    5043 cpu_manager.go:414] "containerMap: removing container" podUID="53142e30-10c0-413a-ae61-fb1b29f5fd10" containerName="istio-proxy"
Oct 30 19:44:26.263105 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:26.263093    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53142e30-10c0-413a-ae61-fb1b29f5fd10" containerName="istio-proxy"
Oct 30 19:44:26.263310 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:26.263134    5043 cpu_manager.go:414] "containerMap: removing container" podUID="377f3c78-650d-4680-aa0f-0373eba791eb" containerName="istio-proxy"
Oct 30 19:44:26.263310 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:26.263143    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="377f3c78-650d-4680-aa0f-0373eba791eb" containerName="istio-proxy"
Oct 30 19:44:26.263310 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:26.263173    5043 cpu_manager.go:414] "containerMap: removing container" podUID="5ff1923b-6547-4e9b-8810-3f79648e3272" containerName="istio-proxy"
Oct 30 19:44:26.263310 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:26.263181    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ff1923b-6547-4e9b-8810-3f79648e3272" containerName="istio-proxy"
Oct 30 19:44:26.263310 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:26.263224    5043 cpu_manager.go:414] "containerMap: removing container" podUID="c5c180ba-4a6f-4fb5-9976-1bd4b3526081" containerName="istio-proxy"
Oct 30 19:44:26.263310 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:26.263237    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5c180ba-4a6f-4fb5-9976-1bd4b3526081" containerName="istio-proxy"
Oct 30 19:44:26.263310 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:26.263256    5043 cpu_manager.go:414] "containerMap: removing container" podUID="9e417b0f-a2b8-4792-9733-76b3a97a33df" containerName="revoke-keys"
Oct 30 19:44:26.263310 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:26.263266    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e417b0f-a2b8-4792-9733-76b3a97a33df" containerName="revoke-keys"
Oct 30 19:44:26.263310 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:26.263295    5043 cpu_manager.go:414] "containerMap: removing container" podUID="5ff1923b-6547-4e9b-8810-3f79648e3272" containerName="istio-proxy"
Oct 30 19:44:26.263310 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:26.263302    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ff1923b-6547-4e9b-8810-3f79648e3272" containerName="istio-proxy"
Oct 30 19:44:26.263834 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:26.263688    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ff1923b-6547-4e9b-8810-3f79648e3272" containerName="istio-proxy"
Oct 30 19:44:26.263834 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:26.263713    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5c180ba-4a6f-4fb5-9976-1bd4b3526081" containerName="istio-proxy"
Oct 30 19:44:26.263834 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:26.263727    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e417b0f-a2b8-4792-9733-76b3a97a33df" containerName="revoke-keys"
Oct 30 19:44:26.263834 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:26.263760    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ff1923b-6547-4e9b-8810-3f79648e3272" containerName="istio-proxy"
Oct 30 19:44:26.263834 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:26.263769    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="53142e30-10c0-413a-ae61-fb1b29f5fd10" containerName="istio-proxy"
Oct 30 19:44:26.263834 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:26.263782    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="377f3c78-650d-4680-aa0f-0373eba791eb" containerName="istio-proxy"
Oct 30 19:44:26.270640 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:26.270617    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-01b24e-9gkpt"
Oct 30 19:44:26.280786 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:26.279340    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:44:26.286842 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:26.285448    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:44:26.288566 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:26.288369    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-01b24e-9gkpt"]
Oct 30 19:44:26.296819 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:26.293967    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:44:26.296819 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:26.294041    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:44:26.430406 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:26.430354    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00c8d4cc-34b3-4022-8434-f41d5bd176ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-01b24e-9gkpt\" (UID: \"00c8d4cc-34b3-4022-8434-f41d5bd176ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-01b24e-9gkpt"
Oct 30 19:44:26.430605 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:26.430492    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00c8d4cc-34b3-4022-8434-f41d5bd176ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-01b24e-9gkpt\" (UID: \"00c8d4cc-34b3-4022-8434-f41d5bd176ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-01b24e-9gkpt"
Oct 30 19:44:26.430605 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:26.430527    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpf68\" (UniqueName: \"kubernetes.io/projected/00c8d4cc-34b3-4022-8434-f41d5bd176ed-kube-api-access-vpf68\") pod \"maas-api-revoke-keys-e2e-worker-w6-01b24e-9gkpt\" (UID: \"00c8d4cc-34b3-4022-8434-f41d5bd176ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-01b24e-9gkpt"
Oct 30 19:44:26.479559 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:26.479518    5043 patch_prober.go:29] interesting pod/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:44:26.479728 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:26.479581    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx" podUID="f791cec1-a1e4-4067-9221-cc04dafba0eb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:44:26.532315 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:26.532238    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00c8d4cc-34b3-4022-8434-f41d5bd176ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-01b24e-9gkpt\" (UID: \"00c8d4cc-34b3-4022-8434-f41d5bd176ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-01b24e-9gkpt"
Oct 30 19:44:26.532315 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:26.532301    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00c8d4cc-34b3-4022-8434-f41d5bd176ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-01b24e-9gkpt\" (UID: \"00c8d4cc-34b3-4022-8434-f41d5bd176ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-01b24e-9gkpt"
Oct 30 19:44:26.532531 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:26.532320    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpf68\" (UniqueName: \"kubernetes.io/projected/00c8d4cc-34b3-4022-8434-f41d5bd176ed-kube-api-access-vpf68\") pod \"maas-api-revoke-keys-e2e-worker-w6-01b24e-9gkpt\" (UID: \"00c8d4cc-34b3-4022-8434-f41d5bd176ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-01b24e-9gkpt"
Oct 30 19:44:26.533053 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:26.533029    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00c8d4cc-34b3-4022-8434-f41d5bd176ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-01b24e-9gkpt\" (UID: \"00c8d4cc-34b3-4022-8434-f41d5bd176ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-01b24e-9gkpt"
Oct 30 19:44:26.534706 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:26.534687    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00c8d4cc-34b3-4022-8434-f41d5bd176ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-01b24e-9gkpt\" (UID: \"00c8d4cc-34b3-4022-8434-f41d5bd176ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-01b24e-9gkpt"
Oct 30 19:44:26.550414 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:26.550392    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpf68\" (UniqueName: \"kubernetes.io/projected/00c8d4cc-34b3-4022-8434-f41d5bd176ed-kube-api-access-vpf68\") pod \"maas-api-revoke-keys-e2e-worker-w6-01b24e-9gkpt\" (UID: \"00c8d4cc-34b3-4022-8434-f41d5bd176ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-01b24e-9gkpt"
Oct 30 19:44:26.624636 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:26.624605    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-01b24e-9gkpt"
Oct 30 19:44:26.920993 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:26.920959    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-01b24e-9gkpt"]
Oct 30 19:44:26.921923 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:44:26.921895    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00c8d4cc_34b3_4022_8434_f41d5bd176ed.slice/crio-8f97fe1112bcdad070d9b52775ee9309ff8cc5a5d19eb26690caef82a2094c0d WatchSource:0}: Error finding container 8f97fe1112bcdad070d9b52775ee9309ff8cc5a5d19eb26690caef82a2094c0d: Status 404 returned error can't find the container with id 8f97fe1112bcdad070d9b52775ee9309ff8cc5a5d19eb26690caef82a2094c0d
Oct 30 19:44:27.461731 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:27.461689    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-01b24e-9gkpt" event={"ID":"00c8d4cc-34b3-4022-8434-f41d5bd176ed","Type":"ContainerStarted","Data":"9c571999adeb6a26ef885dd09fc1048a7cf31d24b4e7351856332a2b48460a53"}
Oct 30 19:44:27.462241 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:27.461821    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-01b24e-9gkpt" event={"ID":"00c8d4cc-34b3-4022-8434-f41d5bd176ed","Type":"ContainerStarted","Data":"8f97fe1112bcdad070d9b52775ee9309ff8cc5a5d19eb26690caef82a2094c0d"}
Oct 30 19:44:27.490540 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:27.490479    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-01b24e-9gkpt" podStartSLOduration=1.490463186 podStartE2EDuration="1.490463186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:27.489796695 +0000 UTC m=+1687.000856660" watchObservedRunningTime="2026-10-30 19:44:27.490463186 +0000 UTC m=+1687.001523152"
Oct 30 19:44:28.474248 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:28.474212    5043 generic.go:356] "Generic (PLEG): container finished" podID="00c8d4cc-34b3-4022-8434-f41d5bd176ed" containerID="9c571999adeb6a26ef885dd09fc1048a7cf31d24b4e7351856332a2b48460a53" exitCode=0
Oct 30 19:44:28.474725 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:28.474309    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-01b24e-9gkpt" event={"ID":"00c8d4cc-34b3-4022-8434-f41d5bd176ed","Type":"ContainerDied","Data":"9c571999adeb6a26ef885dd09fc1048a7cf31d24b4e7351856332a2b48460a53"}
Oct 30 19:44:28.521926 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:28.521731    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ac37c8-69dbd9fb6d-xjzdf"]
Oct 30 19:44:28.533850 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:28.533820    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ac37c8-57cb47d8c-4cprd"]
Oct 30 19:44:29.283975 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:29.283924    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj"]
Oct 30 19:44:29.284204 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:29.284173    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj" podUID="097a37e6-fa47-4514-b43d-5c462533947b" containerName="istio-proxy" containerID="cri-o://0d8116adb542a69bb69ceaba30af99c1caedb9f5dacf472d117b787d3714c39b" gracePeriod=30
Oct 30 19:44:29.674797 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:29.674377    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89a3f6-db46ffd87-8l4s5" podUID="62c3de5f-9601-43f4-a89a-85c623e1222b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:44:29.682836 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:29.678524    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-01b24e-9gkpt"
Oct 30 19:44:29.781044 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:29.781012    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00c8d4cc-34b3-4022-8434-f41d5bd176ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00c8d4cc-34b3-4022-8434-f41d5bd176ed-maas-api-xks-ca\") pod \"00c8d4cc-34b3-4022-8434-f41d5bd176ed\" (UID: \"00c8d4cc-34b3-4022-8434-f41d5bd176ed\") "
Oct 30 19:44:29.781174 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:29.781070    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00c8d4cc-34b3-4022-8434-f41d5bd176ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00c8d4cc-34b3-4022-8434-f41d5bd176ed-maas-api-openshift-ca\") pod \"00c8d4cc-34b3-4022-8434-f41d5bd176ed\" (UID: \"00c8d4cc-34b3-4022-8434-f41d5bd176ed\") "
Oct 30 19:44:29.781244 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:29.781226    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00c8d4cc-34b3-4022-8434-f41d5bd176ed-kube-api-access-vpf68\" (UniqueName: \"kubernetes.io/projected/00c8d4cc-34b3-4022-8434-f41d5bd176ed-kube-api-access-vpf68\") pod \"00c8d4cc-34b3-4022-8434-f41d5bd176ed\" (UID: \"00c8d4cc-34b3-4022-8434-f41d5bd176ed\") "
Oct 30 19:44:29.781622 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:29.781563    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00c8d4cc-34b3-4022-8434-f41d5bd176ed-maas-api-openshift-ca" pod "00c8d4cc-34b3-4022-8434-f41d5bd176ed" (UID: "00c8d4cc-34b3-4022-8434-f41d5bd176ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:29.781937 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:29.781920    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00c8d4cc-34b3-4022-8434-f41d5bd176ed-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:29.783968 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:29.783947    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00c8d4cc-34b3-4022-8434-f41d5bd176ed-kube-api-access-vpf68" pod "00c8d4cc-34b3-4022-8434-f41d5bd176ed" (UID: "00c8d4cc-34b3-4022-8434-f41d5bd176ed"). InnerVolumeSpecName "kube-api-access-vpf68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:29.787464 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:29.787437    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00c8d4cc-34b3-4022-8434-f41d5bd176ed-maas-api-xks-ca" pod "00c8d4cc-34b3-4022-8434-f41d5bd176ed" (UID: "00c8d4cc-34b3-4022-8434-f41d5bd176ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:44:29.811956 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:29.811906    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1e13df-78b7ccf4bc-8ldn2"
Oct 30 19:44:29.846880 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:29.846846    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d651c4-ww6bm"]
Oct 30 19:44:29.847464 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:29.847446    5043 cpu_manager.go:414] "containerMap: removing container" podUID="3cd1e9e6-6a23-40d8-aa51-0266d9c61446" containerName="payload-pre-processing"
Oct 30 19:44:29.847606 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:29.847465    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cd1e9e6-6a23-40d8-aa51-0266d9c61446" containerName="payload-pre-processing"
Oct 30 19:44:29.847606 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:29.847487    5043 cpu_manager.go:414] "containerMap: removing container" podUID="00c8d4cc-34b3-4022-8434-f41d5bd176ed" containerName="revoke-keys"
Oct 30 19:44:29.847606 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:29.847492    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00c8d4cc-34b3-4022-8434-f41d5bd176ed" containerName="revoke-keys"
Oct 30 19:44:29.847799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:29.847621    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cd1e9e6-6a23-40d8-aa51-0266d9c61446" containerName="payload-pre-processing"
Oct 30 19:44:29.847799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:29.847632    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="00c8d4cc-34b3-4022-8434-f41d5bd176ed" containerName="revoke-keys"
Oct 30 19:44:29.851276 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:29.851253    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d651c4-ww6bm"
Oct 30 19:44:29.874323 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:29.874037    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d651c4-ww6bm"]
Oct 30 19:44:29.884503 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:29.884471    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00c8d4cc-34b3-4022-8434-f41d5bd176ed-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:29.884664 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:29.884506    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpf68\" (UniqueName: \"kubernetes.io/projected/00c8d4cc-34b3-4022-8434-f41d5bd176ed-kube-api-access-vpf68\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:29.985971 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:29.985936    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cd1e9e6-6a23-40d8-aa51-0266d9c61446-kube-api-access-bhrbb\" (UniqueName: \"kubernetes.io/projected/3cd1e9e6-6a23-40d8-aa51-0266d9c61446-kube-api-access-bhrbb\") pod \"3cd1e9e6-6a23-40d8-aa51-0266d9c61446\" (UID: \"3cd1e9e6-6a23-40d8-aa51-0266d9c61446\") "
Oct 30 19:44:29.986137 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:29.986027    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cd1e9e6-6a23-40d8-aa51-0266d9c61446-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cd1e9e6-6a23-40d8-aa51-0266d9c61446-plugins-config-volume\") pod \"3cd1e9e6-6a23-40d8-aa51-0266d9c61446\" (UID: \"3cd1e9e6-6a23-40d8-aa51-0266d9c61446\") "
Oct 30 19:44:29.986348 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:29.986329    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67lgd\" (UniqueName: \"kubernetes.io/projected/edc81663-c247-4c5e-8d01-89aa8f804165-kube-api-access-67lgd\") pod \"maas-api-revoke-keys-e2e-inf-b-d651c4-ww6bm\" (UID: \"edc81663-c247-4c5e-8d01-89aa8f804165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d651c4-ww6bm"
Oct 30 19:44:29.986453 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:29.986410    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cd1e9e6-6a23-40d8-aa51-0266d9c61446-plugins-config-volume" pod "3cd1e9e6-6a23-40d8-aa51-0266d9c61446" (UID: "3cd1e9e6-6a23-40d8-aa51-0266d9c61446"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:29.986526 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:29.986477    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edc81663-c247-4c5e-8d01-89aa8f804165-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d651c4-ww6bm\" (UID: \"edc81663-c247-4c5e-8d01-89aa8f804165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d651c4-ww6bm"
Oct 30 19:44:29.986582 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:29.986531    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edc81663-c247-4c5e-8d01-89aa8f804165-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d651c4-ww6bm\" (UID: \"edc81663-c247-4c5e-8d01-89aa8f804165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d651c4-ww6bm"
Oct 30 19:44:29.986651 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:29.986637    5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cd1e9e6-6a23-40d8-aa51-0266d9c61446-plugins-config-volume\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:29.988224 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:29.988191    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cd1e9e6-6a23-40d8-aa51-0266d9c61446-kube-api-access-bhrbb" pod "3cd1e9e6-6a23-40d8-aa51-0266d9c61446" (UID: "3cd1e9e6-6a23-40d8-aa51-0266d9c61446"). InnerVolumeSpecName "kube-api-access-bhrbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:30.087951 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:30.087848    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edc81663-c247-4c5e-8d01-89aa8f804165-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d651c4-ww6bm\" (UID: \"edc81663-c247-4c5e-8d01-89aa8f804165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d651c4-ww6bm"
Oct 30 19:44:30.087951 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:30.087919    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edc81663-c247-4c5e-8d01-89aa8f804165-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d651c4-ww6bm\" (UID: \"edc81663-c247-4c5e-8d01-89aa8f804165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d651c4-ww6bm"
Oct 30 19:44:30.088209 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:30.088092    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67lgd\" (UniqueName: \"kubernetes.io/projected/edc81663-c247-4c5e-8d01-89aa8f804165-kube-api-access-67lgd\") pod \"maas-api-revoke-keys-e2e-inf-b-d651c4-ww6bm\" (UID: \"edc81663-c247-4c5e-8d01-89aa8f804165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d651c4-ww6bm"
Oct 30 19:44:30.088209 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:30.088186    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhrbb\" (UniqueName: \"kubernetes.io/projected/3cd1e9e6-6a23-40d8-aa51-0266d9c61446-kube-api-access-bhrbb\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:30.088640 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:30.088616    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edc81663-c247-4c5e-8d01-89aa8f804165-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d651c4-ww6bm\" (UID: \"edc81663-c247-4c5e-8d01-89aa8f804165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d651c4-ww6bm"
Oct 30 19:44:30.090521 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:30.090497    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edc81663-c247-4c5e-8d01-89aa8f804165-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d651c4-ww6bm\" (UID: \"edc81663-c247-4c5e-8d01-89aa8f804165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d651c4-ww6bm"
Oct 30 19:44:30.104354 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:30.104328    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67lgd\" (UniqueName: \"kubernetes.io/projected/edc81663-c247-4c5e-8d01-89aa8f804165-kube-api-access-67lgd\") pod \"maas-api-revoke-keys-e2e-inf-b-d651c4-ww6bm\" (UID: \"edc81663-c247-4c5e-8d01-89aa8f804165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d651c4-ww6bm"
Oct 30 19:44:30.195695 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:30.195659    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d651c4-ww6bm"
Oct 30 19:44:30.497215 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:30.497184    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d651c4-ww6bm"]
Oct 30 19:44:30.497410 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:30.497387    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-01b24e-9gkpt"
Oct 30 19:44:30.497482 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:30.497396    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-01b24e-9gkpt" event={"ID":"00c8d4cc-34b3-4022-8434-f41d5bd176ed","Type":"ContainerDied","Data":"8f97fe1112bcdad070d9b52775ee9309ff8cc5a5d19eb26690caef82a2094c0d"}
Oct 30 19:44:30.497482 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:30.497452    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f97fe1112bcdad070d9b52775ee9309ff8cc5a5d19eb26690caef82a2094c0d"
Oct 30 19:44:30.498234 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:44:30.498210    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedc81663_c247_4c5e_8d01_89aa8f804165.slice/crio-6180aba46f30db8684438cda52416266771c64bb1252d419dd418042df62e184 WatchSource:0}: Error finding container 6180aba46f30db8684438cda52416266771c64bb1252d419dd418042df62e184: Status 404 returned error can't find the container with id 6180aba46f30db8684438cda52416266771c64bb1252d419dd418042df62e184
Oct 30 19:44:30.499039 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:30.499014    5043 generic.go:356] "Generic (PLEG): container finished" podID="3cd1e9e6-6a23-40d8-aa51-0266d9c61446" containerID="a214ad1cb247e0cbbae5d35dd948109e53bdc09f70b4d3c45c691db8f7ec9d41" exitCode=0
Oct 30 19:44:30.499154 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:30.499064    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1e13df-78b7ccf4bc-8ldn2"
Oct 30 19:44:30.499283 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:30.499210    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1e13df-78b7ccf4bc-8ldn2" event={"ID":"3cd1e9e6-6a23-40d8-aa51-0266d9c61446","Type":"ContainerDied","Data":"a214ad1cb247e0cbbae5d35dd948109e53bdc09f70b4d3c45c691db8f7ec9d41"}
Oct 30 19:44:30.499283 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:30.499251    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1e13df-78b7ccf4bc-8ldn2" event={"ID":"3cd1e9e6-6a23-40d8-aa51-0266d9c61446","Type":"ContainerDied","Data":"d6f00b150c12a7505e8d9ef27ad9441cbe92df189fd2494e7e0074b3971332a4"}
Oct 30 19:44:30.499283 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:30.499279    5043 scope.go:122] "RemoveContainer" containerID="a214ad1cb247e0cbbae5d35dd948109e53bdc09f70b4d3c45c691db8f7ec9d41"
Oct 30 19:44:30.608209 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:30.608186    5043 scope.go:122] "RemoveContainer" containerID="a214ad1cb247e0cbbae5d35dd948109e53bdc09f70b4d3c45c691db8f7ec9d41"
Oct 30 19:44:30.609266 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:44:30.609237    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a214ad1cb247e0cbbae5d35dd948109e53bdc09f70b4d3c45c691db8f7ec9d41\": container with ID starting with a214ad1cb247e0cbbae5d35dd948109e53bdc09f70b4d3c45c691db8f7ec9d41 not found: ID does not exist" containerID="a214ad1cb247e0cbbae5d35dd948109e53bdc09f70b4d3c45c691db8f7ec9d41"
Oct 30 19:44:30.609376 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:30.609272    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a214ad1cb247e0cbbae5d35dd948109e53bdc09f70b4d3c45c691db8f7ec9d41"} err="failed to get container status \"a214ad1cb247e0cbbae5d35dd948109e53bdc09f70b4d3c45c691db8f7ec9d41\": rpc error: code = NotFound desc = could not find container \"a214ad1cb247e0cbbae5d35dd948109e53bdc09f70b4d3c45c691db8f7ec9d41\": container with ID starting with a214ad1cb247e0cbbae5d35dd948109e53bdc09f70b4d3c45c691db8f7ec9d41 not found: ID does not exist"
Oct 30 19:44:30.626347 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:30.626317    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1e13df-78b7ccf4bc-8ldn2"]
Oct 30 19:44:30.628472 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:30.628448    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1e13df-78b7ccf4bc-8ldn2"]
Oct 30 19:44:30.648136 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:30.647795    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cd1e9e6-6a23-40d8-aa51-0266d9c61446" path="/var/lib/kubelet/pods/3cd1e9e6-6a23-40d8-aa51-0266d9c61446/volumes"
Oct 30 19:44:30.820285 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:30.819913    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-744b57-77cc66bfc9-2fcz9"
Oct 30 19:44:30.850240 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:30.850213    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx"
Oct 30 19:44:31.009813 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.009780    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ef41111-49f3-4fd9-9587-2a1934f50797-kube-api-access-xwqtn\" (UniqueName: \"kubernetes.io/projected/4ef41111-49f3-4fd9-9587-2a1934f50797-kube-api-access-xwqtn\") pod \"4ef41111-49f3-4fd9-9587-2a1934f50797\" (UID: \"4ef41111-49f3-4fd9-9587-2a1934f50797\") "
Oct 30 19:44:31.009956 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.009858    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f791cec1-a1e4-4067-9221-cc04dafba0eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f791cec1-a1e4-4067-9221-cc04dafba0eb-istio-data\") pod \"f791cec1-a1e4-4067-9221-cc04dafba0eb\" (UID: \"f791cec1-a1e4-4067-9221-cc04dafba0eb\") "
Oct 30 19:44:31.009956 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.009886    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f791cec1-a1e4-4067-9221-cc04dafba0eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f791cec1-a1e4-4067-9221-cc04dafba0eb-credential-socket\") pod \"f791cec1-a1e4-4067-9221-cc04dafba0eb\" (UID: \"f791cec1-a1e4-4067-9221-cc04dafba0eb\") "
Oct 30 19:44:31.009956 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.009940    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f791cec1-a1e4-4067-9221-cc04dafba0eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f791cec1-a1e4-4067-9221-cc04dafba0eb-workload-certs\") pod \"f791cec1-a1e4-4067-9221-cc04dafba0eb\" (UID: \"f791cec1-a1e4-4067-9221-cc04dafba0eb\") "
Oct 30 19:44:31.010089 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.010047    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f791cec1-a1e4-4067-9221-cc04dafba0eb-istio-token\" (UniqueName: \"kubernetes.io/projected/f791cec1-a1e4-4067-9221-cc04dafba0eb-istio-token\") pod \"f791cec1-a1e4-4067-9221-cc04dafba0eb\" (UID: \"f791cec1-a1e4-4067-9221-cc04dafba0eb\") "
Oct 30 19:44:31.010177 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.010092    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f791cec1-a1e4-4067-9221-cc04dafba0eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f791cec1-a1e4-4067-9221-cc04dafba0eb-istio-podinfo\") pod \"f791cec1-a1e4-4067-9221-cc04dafba0eb\" (UID: \"f791cec1-a1e4-4067-9221-cc04dafba0eb\") "
Oct 30 19:44:31.010612 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.010327    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f791cec1-a1e4-4067-9221-cc04dafba0eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f791cec1-a1e4-4067-9221-cc04dafba0eb-workload-socket\") pod \"f791cec1-a1e4-4067-9221-cc04dafba0eb\" (UID: \"f791cec1-a1e4-4067-9221-cc04dafba0eb\") "
Oct 30 19:44:31.010612 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.010312    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f791cec1-a1e4-4067-9221-cc04dafba0eb-workload-certs" pod "f791cec1-a1e4-4067-9221-cc04dafba0eb" (UID: "f791cec1-a1e4-4067-9221-cc04dafba0eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:31.010612 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.010365    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f791cec1-a1e4-4067-9221-cc04dafba0eb-credential-socket" pod "f791cec1-a1e4-4067-9221-cc04dafba0eb" (UID: "f791cec1-a1e4-4067-9221-cc04dafba0eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:31.010612 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.010409    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f791cec1-a1e4-4067-9221-cc04dafba0eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f791cec1-a1e4-4067-9221-cc04dafba0eb-istio-envoy\") pod \"f791cec1-a1e4-4067-9221-cc04dafba0eb\" (UID: \"f791cec1-a1e4-4067-9221-cc04dafba0eb\") "
Oct 30 19:44:31.010612 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.010482    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f791cec1-a1e4-4067-9221-cc04dafba0eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f791cec1-a1e4-4067-9221-cc04dafba0eb-istiod-ca-cert\") pod \"f791cec1-a1e4-4067-9221-cc04dafba0eb\" (UID: \"f791cec1-a1e4-4067-9221-cc04dafba0eb\") "
Oct 30 19:44:31.010612 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.010565    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f791cec1-a1e4-4067-9221-cc04dafba0eb-workload-socket" pod "f791cec1-a1e4-4067-9221-cc04dafba0eb" (UID: "f791cec1-a1e4-4067-9221-cc04dafba0eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:31.011590 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.010627    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f791cec1-a1e4-4067-9221-cc04dafba0eb-kube-api-access-899wx\" (UniqueName: \"kubernetes.io/projected/f791cec1-a1e4-4067-9221-cc04dafba0eb-kube-api-access-899wx\") pod \"f791cec1-a1e4-4067-9221-cc04dafba0eb\" (UID: \"f791cec1-a1e4-4067-9221-cc04dafba0eb\") "
Oct 30 19:44:31.011590 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.010774    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ef41111-49f3-4fd9-9587-2a1934f50797-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ef41111-49f3-4fd9-9587-2a1934f50797-plugins-config-volume\") pod \"4ef41111-49f3-4fd9-9587-2a1934f50797\" (UID: \"4ef41111-49f3-4fd9-9587-2a1934f50797\") "
Oct 30 19:44:31.011590 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.011023    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f791cec1-a1e4-4067-9221-cc04dafba0eb-istiod-ca-cert" pod "f791cec1-a1e4-4067-9221-cc04dafba0eb" (UID: "f791cec1-a1e4-4067-9221-cc04dafba0eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:31.011590 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.011346    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f791cec1-a1e4-4067-9221-cc04dafba0eb-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:31.011590 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.011367    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f791cec1-a1e4-4067-9221-cc04dafba0eb-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:31.011590 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.011381    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f791cec1-a1e4-4067-9221-cc04dafba0eb-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:31.011590 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.011394    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f791cec1-a1e4-4067-9221-cc04dafba0eb-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:31.011590 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.011574    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ef41111-49f3-4fd9-9587-2a1934f50797-plugins-config-volume" pod "4ef41111-49f3-4fd9-9587-2a1934f50797" (UID: "4ef41111-49f3-4fd9-9587-2a1934f50797"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:31.012721 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.012084    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f791cec1-a1e4-4067-9221-cc04dafba0eb-istio-data" pod "f791cec1-a1e4-4067-9221-cc04dafba0eb" (UID: "f791cec1-a1e4-4067-9221-cc04dafba0eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:31.013511 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.013488    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f791cec1-a1e4-4067-9221-cc04dafba0eb-istio-envoy" pod "f791cec1-a1e4-4067-9221-cc04dafba0eb" (UID: "f791cec1-a1e4-4067-9221-cc04dafba0eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:31.013841 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.013821    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f791cec1-a1e4-4067-9221-cc04dafba0eb-istio-podinfo" pod "f791cec1-a1e4-4067-9221-cc04dafba0eb" (UID: "f791cec1-a1e4-4067-9221-cc04dafba0eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:44:31.013949 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.013927    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ef41111-49f3-4fd9-9587-2a1934f50797-kube-api-access-xwqtn" pod "4ef41111-49f3-4fd9-9587-2a1934f50797" (UID: "4ef41111-49f3-4fd9-9587-2a1934f50797"). InnerVolumeSpecName "kube-api-access-xwqtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:31.014325 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.014307    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f791cec1-a1e4-4067-9221-cc04dafba0eb-kube-api-access-899wx" pod "f791cec1-a1e4-4067-9221-cc04dafba0eb" (UID: "f791cec1-a1e4-4067-9221-cc04dafba0eb"). InnerVolumeSpecName "kube-api-access-899wx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:31.014874 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.014857    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f791cec1-a1e4-4067-9221-cc04dafba0eb-istio-token" pod "f791cec1-a1e4-4067-9221-cc04dafba0eb" (UID: "f791cec1-a1e4-4067-9221-cc04dafba0eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:31.039747 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.039667    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fd3f959-sv82j"]
Oct 30 19:44:31.043088 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.043066    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6970fe37-h6trh"]
Oct 30 19:44:31.047286 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.047258    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fd3f959-sv82j"]
Oct 30 19:44:31.068641 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.068606    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"]
Oct 30 19:44:31.072516 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.072487    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6970fe37-h6trh"]
Oct 30 19:44:31.102599 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.102567    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"]
Oct 30 19:44:31.112257 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.112213    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f791cec1-a1e4-4067-9221-cc04dafba0eb-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:31.112257 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.112237    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f791cec1-a1e4-4067-9221-cc04dafba0eb-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:31.112257 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.112248    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f791cec1-a1e4-4067-9221-cc04dafba0eb-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:31.112257 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.112257    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-899wx\" (UniqueName: \"kubernetes.io/projected/f791cec1-a1e4-4067-9221-cc04dafba0eb-kube-api-access-899wx\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:31.112545 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.112268    5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ef41111-49f3-4fd9-9587-2a1934f50797-plugins-config-volume\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:31.112545 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.112281    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwqtn\" (UniqueName: \"kubernetes.io/projected/4ef41111-49f3-4fd9-9587-2a1934f50797-kube-api-access-xwqtn\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:31.112545 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.112294    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f791cec1-a1e4-4067-9221-cc04dafba0eb-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:31.508281 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.508245    5043 generic.go:356] "Generic (PLEG): container finished" podID="4ef41111-49f3-4fd9-9587-2a1934f50797" containerID="af0969d1dcc550e1c71ac468872d2556fec54bb81e7596638a935a164561936c" exitCode=0
Oct 30 19:44:31.508536 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.508299    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-744b57-77cc66bfc9-2fcz9"
Oct 30 19:44:31.508536 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.508295    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-744b57-77cc66bfc9-2fcz9" event={"ID":"4ef41111-49f3-4fd9-9587-2a1934f50797","Type":"ContainerDied","Data":"af0969d1dcc550e1c71ac468872d2556fec54bb81e7596638a935a164561936c"}
Oct 30 19:44:31.508536 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.508355    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-744b57-77cc66bfc9-2fcz9" event={"ID":"4ef41111-49f3-4fd9-9587-2a1934f50797","Type":"ContainerDied","Data":"f9d9e69c15cc1f0edc4a4f95d4b43647b87e9f559d96736226c9f3224b8d69bc"}
Oct 30 19:44:31.508536 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.508384    5043 scope.go:122] "RemoveContainer" containerID="af0969d1dcc550e1c71ac468872d2556fec54bb81e7596638a935a164561936c"
Oct 30 19:44:31.511934 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.511904    5043 generic.go:356] "Generic (PLEG): container finished" podID="f791cec1-a1e4-4067-9221-cc04dafba0eb" containerID="cb07df21823bf0822cc28be90ffa63cfdc67e892be4a904cc079f6e990725195" exitCode=0
Oct 30 19:44:31.512054 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.511953    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx"
Oct 30 19:44:31.512054 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.511958    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx" event={"ID":"f791cec1-a1e4-4067-9221-cc04dafba0eb","Type":"ContainerDied","Data":"cb07df21823bf0822cc28be90ffa63cfdc67e892be4a904cc079f6e990725195"}
Oct 30 19:44:31.512054 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.512017    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx" event={"ID":"f791cec1-a1e4-4067-9221-cc04dafba0eb","Type":"ContainerDied","Data":"f9741431c94b72b493449574cba873714903d28bd849f67b1fcc40a0022c2ffa"}
Oct 30 19:44:31.513915 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.513889    5043 generic.go:356] "Generic (PLEG): container finished" podID="edc81663-c247-4c5e-8d01-89aa8f804165" containerID="7c93c003c43c9f3ffd8983e52e6768d9f1532cc7edcd688c728e86b782725820" exitCode=0
Oct 30 19:44:31.514030 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.513944    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d651c4-ww6bm" event={"ID":"edc81663-c247-4c5e-8d01-89aa8f804165","Type":"ContainerDied","Data":"7c93c003c43c9f3ffd8983e52e6768d9f1532cc7edcd688c728e86b782725820"}
Oct 30 19:44:31.514030 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.513989    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d651c4-ww6bm" event={"ID":"edc81663-c247-4c5e-8d01-89aa8f804165","Type":"ContainerStarted","Data":"6180aba46f30db8684438cda52416266771c64bb1252d419dd418042df62e184"}
Oct 30 19:44:31.528910 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.528890    5043 scope.go:122] "RemoveContainer" containerID="af0969d1dcc550e1c71ac468872d2556fec54bb81e7596638a935a164561936c"
Oct 30 19:44:31.529256 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:44:31.529230    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af0969d1dcc550e1c71ac468872d2556fec54bb81e7596638a935a164561936c\": container with ID starting with af0969d1dcc550e1c71ac468872d2556fec54bb81e7596638a935a164561936c not found: ID does not exist" containerID="af0969d1dcc550e1c71ac468872d2556fec54bb81e7596638a935a164561936c"
Oct 30 19:44:31.529377 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.529264    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af0969d1dcc550e1c71ac468872d2556fec54bb81e7596638a935a164561936c"} err="failed to get container status \"af0969d1dcc550e1c71ac468872d2556fec54bb81e7596638a935a164561936c\": rpc error: code = NotFound desc = could not find container \"af0969d1dcc550e1c71ac468872d2556fec54bb81e7596638a935a164561936c\": container with ID starting with af0969d1dcc550e1c71ac468872d2556fec54bb81e7596638a935a164561936c not found: ID does not exist"
Oct 30 19:44:31.529377 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.529281    5043 scope.go:122] "RemoveContainer" containerID="cb07df21823bf0822cc28be90ffa63cfdc67e892be4a904cc079f6e990725195"
Oct 30 19:44:31.552929 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.552902    5043 scope.go:122] "RemoveContainer" containerID="2bf0d118cce4ea36dc19bc4338eaf9c3cf4336991c115f7aa305eedaa4b76956"
Oct 30 19:44:31.560911 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.560861    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx"]
Oct 30 19:44:31.566605 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.566577    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-c6zvx"]
Oct 30 19:44:31.567042 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.567013    5043 scope.go:122] "RemoveContainer" containerID="cb07df21823bf0822cc28be90ffa63cfdc67e892be4a904cc079f6e990725195"
Oct 30 19:44:31.567350 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:44:31.567329    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb07df21823bf0822cc28be90ffa63cfdc67e892be4a904cc079f6e990725195\": container with ID starting with cb07df21823bf0822cc28be90ffa63cfdc67e892be4a904cc079f6e990725195 not found: ID does not exist" containerID="cb07df21823bf0822cc28be90ffa63cfdc67e892be4a904cc079f6e990725195"
Oct 30 19:44:31.567420 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.567363    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb07df21823bf0822cc28be90ffa63cfdc67e892be4a904cc079f6e990725195"} err="failed to get container status \"cb07df21823bf0822cc28be90ffa63cfdc67e892be4a904cc079f6e990725195\": rpc error: code = NotFound desc = could not find container \"cb07df21823bf0822cc28be90ffa63cfdc67e892be4a904cc079f6e990725195\": container with ID starting with cb07df21823bf0822cc28be90ffa63cfdc67e892be4a904cc079f6e990725195 not found: ID does not exist"
Oct 30 19:44:31.567420 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.567387    5043 scope.go:122] "RemoveContainer" containerID="2bf0d118cce4ea36dc19bc4338eaf9c3cf4336991c115f7aa305eedaa4b76956"
Oct 30 19:44:31.567699 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:44:31.567681    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bf0d118cce4ea36dc19bc4338eaf9c3cf4336991c115f7aa305eedaa4b76956\": container with ID starting with 2bf0d118cce4ea36dc19bc4338eaf9c3cf4336991c115f7aa305eedaa4b76956 not found: ID does not exist" containerID="2bf0d118cce4ea36dc19bc4338eaf9c3cf4336991c115f7aa305eedaa4b76956"
Oct 30 19:44:31.567784 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.567705    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bf0d118cce4ea36dc19bc4338eaf9c3cf4336991c115f7aa305eedaa4b76956"} err="failed to get container status \"2bf0d118cce4ea36dc19bc4338eaf9c3cf4336991c115f7aa305eedaa4b76956\": rpc error: code = NotFound desc = could not find container \"2bf0d118cce4ea36dc19bc4338eaf9c3cf4336991c115f7aa305eedaa4b76956\": container with ID starting with 2bf0d118cce4ea36dc19bc4338eaf9c3cf4336991c115f7aa305eedaa4b76956 not found: ID does not exist"
Oct 30 19:44:31.586668 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.586646    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-744b57-77cc66bfc9-2fcz9"]
Oct 30 19:44:31.598009 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:31.597987    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-744b57-77cc66bfc9-2fcz9"]
Oct 30 19:44:32.652487 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:32.652454    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ef41111-49f3-4fd9-9587-2a1934f50797" path="/var/lib/kubelet/pods/4ef41111-49f3-4fd9-9587-2a1934f50797/volumes"
Oct 30 19:44:32.653052 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:32.653013    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98988463-cb27-4b5d-8e96-bd118b70a212" path="/var/lib/kubelet/pods/98988463-cb27-4b5d-8e96-bd118b70a212/volumes"
Oct 30 19:44:32.653623 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:32.653605    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c96f23c-c4db-409e-acde-e723d8fe248c" path="/var/lib/kubelet/pods/9c96f23c-c4db-409e-acde-e723d8fe248c/volumes"
Oct 30 19:44:32.654693 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:32.654675    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f791cec1-a1e4-4067-9221-cc04dafba0eb" path="/var/lib/kubelet/pods/f791cec1-a1e4-4067-9221-cc04dafba0eb/volumes"
Oct 30 19:44:32.708109 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:32.708083    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d651c4-ww6bm"
Oct 30 19:44:32.733576 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:32.733550    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edc81663-c247-4c5e-8d01-89aa8f804165-kube-api-access-67lgd\" (UniqueName: \"kubernetes.io/projected/edc81663-c247-4c5e-8d01-89aa8f804165-kube-api-access-67lgd\") pod \"edc81663-c247-4c5e-8d01-89aa8f804165\" (UID: \"edc81663-c247-4c5e-8d01-89aa8f804165\") "
Oct 30 19:44:32.733790 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:32.733721    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edc81663-c247-4c5e-8d01-89aa8f804165-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edc81663-c247-4c5e-8d01-89aa8f804165-maas-api-xks-ca\") pod \"edc81663-c247-4c5e-8d01-89aa8f804165\" (UID: \"edc81663-c247-4c5e-8d01-89aa8f804165\") "
Oct 30 19:44:32.733952 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:32.733886    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edc81663-c247-4c5e-8d01-89aa8f804165-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edc81663-c247-4c5e-8d01-89aa8f804165-maas-api-openshift-ca\") pod \"edc81663-c247-4c5e-8d01-89aa8f804165\" (UID: \"edc81663-c247-4c5e-8d01-89aa8f804165\") "
Oct 30 19:44:32.734272 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:32.734245    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edc81663-c247-4c5e-8d01-89aa8f804165-maas-api-openshift-ca" pod "edc81663-c247-4c5e-8d01-89aa8f804165" (UID: "edc81663-c247-4c5e-8d01-89aa8f804165"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:32.734534 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:32.734511    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edc81663-c247-4c5e-8d01-89aa8f804165-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:32.736308 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:32.736280    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edc81663-c247-4c5e-8d01-89aa8f804165-kube-api-access-67lgd" pod "edc81663-c247-4c5e-8d01-89aa8f804165" (UID: "edc81663-c247-4c5e-8d01-89aa8f804165"). InnerVolumeSpecName "kube-api-access-67lgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:32.736402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:32.736342    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edc81663-c247-4c5e-8d01-89aa8f804165-maas-api-xks-ca" pod "edc81663-c247-4c5e-8d01-89aa8f804165" (UID: "edc81663-c247-4c5e-8d01-89aa8f804165"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:44:32.835926 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:32.835851    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67lgd\" (UniqueName: \"kubernetes.io/projected/edc81663-c247-4c5e-8d01-89aa8f804165-kube-api-access-67lgd\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:32.835926 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:32.835877    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edc81663-c247-4c5e-8d01-89aa8f804165-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:33.522191 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:33.522153    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-ac37c8-69dbd9fb6d-xjzdf" podUID="1ed7ff0c-6ec2-443f-b1d8-8f7cbf8238ce" containerName="payload-processing" containerID="cri-o://e2b6fc6754c1988efc99e2385cb6d20b4acd29a3db0a2809adaf7febf11279be" gracePeriod=25
Oct 30 19:44:33.534344 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:33.534311    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ac37c8-57cb47d8c-4cprd" podUID="42cfcd85-0bda-4e33-ae8c-f84eb90afd6b" containerName="payload-pre-processing" containerID="cri-o://a85ae4b85ef525013d42625b7cbae6c62a0d89dfa40311c97d4094706e64536e" gracePeriod=25
Oct 30 19:44:33.536666 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:33.536645    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d651c4-ww6bm"
Oct 30 19:44:33.536831 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:33.536675    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d651c4-ww6bm" event={"ID":"edc81663-c247-4c5e-8d01-89aa8f804165","Type":"ContainerDied","Data":"6180aba46f30db8684438cda52416266771c64bb1252d419dd418042df62e184"}
Oct 30 19:44:33.536831 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:33.536750    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6180aba46f30db8684438cda52416266771c64bb1252d419dd418042df62e184"
Oct 30 19:44:33.827698 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:33.827673    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ac37c8-69dbd9fb6d-xjzdf"
Oct 30 19:44:33.851157 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:33.851133    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ed7ff0c-6ec2-443f-b1d8-8f7cbf8238ce-kube-api-access-cwx8b\" (UniqueName: \"kubernetes.io/projected/1ed7ff0c-6ec2-443f-b1d8-8f7cbf8238ce-kube-api-access-cwx8b\") pod \"1ed7ff0c-6ec2-443f-b1d8-8f7cbf8238ce\" (UID: \"1ed7ff0c-6ec2-443f-b1d8-8f7cbf8238ce\") "
Oct 30 19:44:33.851307 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:33.851206    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ed7ff0c-6ec2-443f-b1d8-8f7cbf8238ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ed7ff0c-6ec2-443f-b1d8-8f7cbf8238ce-plugins-config-volume\") pod \"1ed7ff0c-6ec2-443f-b1d8-8f7cbf8238ce\" (UID: \"1ed7ff0c-6ec2-443f-b1d8-8f7cbf8238ce\") "
Oct 30 19:44:33.851687 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:33.851661    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ed7ff0c-6ec2-443f-b1d8-8f7cbf8238ce-plugins-config-volume" pod "1ed7ff0c-6ec2-443f-b1d8-8f7cbf8238ce" (UID: "1ed7ff0c-6ec2-443f-b1d8-8f7cbf8238ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:33.851986 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:33.851832    5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ed7ff0c-6ec2-443f-b1d8-8f7cbf8238ce-plugins-config-volume\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:33.853655 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:33.853630    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ed7ff0c-6ec2-443f-b1d8-8f7cbf8238ce-kube-api-access-cwx8b" pod "1ed7ff0c-6ec2-443f-b1d8-8f7cbf8238ce" (UID: "1ed7ff0c-6ec2-443f-b1d8-8f7cbf8238ce"). InnerVolumeSpecName "kube-api-access-cwx8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:33.938387 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:33.938345    5043 patch_prober.go:29] interesting pod/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:44:33.938530 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:33.938409    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj" podUID="097a37e6-fa47-4514-b43d-5c462533947b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:44:33.953037 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:33.953006    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwx8b\" (UniqueName: \"kubernetes.io/projected/1ed7ff0c-6ec2-443f-b1d8-8f7cbf8238ce-kube-api-access-cwx8b\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:34.550801 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:34.550763    5043 generic.go:356] "Generic (PLEG): container finished" podID="097a37e6-fa47-4514-b43d-5c462533947b" containerID="0d8116adb542a69bb69ceaba30af99c1caedb9f5dacf472d117b787d3714c39b" exitCode=0
Oct 30 19:44:34.550977 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:34.550836    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj" event={"ID":"097a37e6-fa47-4514-b43d-5c462533947b","Type":"ContainerDied","Data":"0d8116adb542a69bb69ceaba30af99c1caedb9f5dacf472d117b787d3714c39b"}
Oct 30 19:44:34.552722 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:34.552697    5043 generic.go:356] "Generic (PLEG): container finished" podID="1ed7ff0c-6ec2-443f-b1d8-8f7cbf8238ce" containerID="e2b6fc6754c1988efc99e2385cb6d20b4acd29a3db0a2809adaf7febf11279be" exitCode=0
Oct 30 19:44:34.552900 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:34.552778    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ac37c8-69dbd9fb6d-xjzdf" event={"ID":"1ed7ff0c-6ec2-443f-b1d8-8f7cbf8238ce","Type":"ContainerDied","Data":"e2b6fc6754c1988efc99e2385cb6d20b4acd29a3db0a2809adaf7febf11279be"}
Oct 30 19:44:34.552900 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:34.552817    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ac37c8-69dbd9fb6d-xjzdf"
Oct 30 19:44:34.552900 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:34.552853    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ac37c8-69dbd9fb6d-xjzdf" event={"ID":"1ed7ff0c-6ec2-443f-b1d8-8f7cbf8238ce","Type":"ContainerDied","Data":"276d32cce14813972e3670ee2a0a093dd88c31ec00f80ba41e619a15628a4c9e"}
Oct 30 19:44:34.552900 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:34.552884    5043 scope.go:122] "RemoveContainer" containerID="e2b6fc6754c1988efc99e2385cb6d20b4acd29a3db0a2809adaf7febf11279be"
Oct 30 19:44:34.573352 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:34.573331    5043 scope.go:122] "RemoveContainer" containerID="e2b6fc6754c1988efc99e2385cb6d20b4acd29a3db0a2809adaf7febf11279be"
Oct 30 19:44:34.573635 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:44:34.573615    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2b6fc6754c1988efc99e2385cb6d20b4acd29a3db0a2809adaf7febf11279be\": container with ID starting with e2b6fc6754c1988efc99e2385cb6d20b4acd29a3db0a2809adaf7febf11279be not found: ID does not exist" containerID="e2b6fc6754c1988efc99e2385cb6d20b4acd29a3db0a2809adaf7febf11279be"
Oct 30 19:44:34.573708 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:34.573647    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2b6fc6754c1988efc99e2385cb6d20b4acd29a3db0a2809adaf7febf11279be"} err="failed to get container status \"e2b6fc6754c1988efc99e2385cb6d20b4acd29a3db0a2809adaf7febf11279be\": rpc error: code = NotFound desc = could not find container \"e2b6fc6754c1988efc99e2385cb6d20b4acd29a3db0a2809adaf7febf11279be\": container with ID starting with e2b6fc6754c1988efc99e2385cb6d20b4acd29a3db0a2809adaf7febf11279be not found: ID does not exist"
Oct 30 19:44:34.577070 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:34.577051    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj"
Oct 30 19:44:34.598902 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:34.598866    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ac37c8-69dbd9fb6d-xjzdf"]
Oct 30 19:44:34.604751 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:34.604705    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ac37c8-69dbd9fb6d-xjzdf"]
Oct 30 19:44:34.650013 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:34.649973    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ed7ff0c-6ec2-443f-b1d8-8f7cbf8238ce" path="/var/lib/kubelet/pods/1ed7ff0c-6ec2-443f-b1d8-8f7cbf8238ce/volumes"
Oct 30 19:44:34.662425 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:34.662396    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/097a37e6-fa47-4514-b43d-5c462533947b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/097a37e6-fa47-4514-b43d-5c462533947b-istio-podinfo\") pod \"097a37e6-fa47-4514-b43d-5c462533947b\" (UID: \"097a37e6-fa47-4514-b43d-5c462533947b\") "
Oct 30 19:44:34.662574 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:34.662451    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/097a37e6-fa47-4514-b43d-5c462533947b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/097a37e6-fa47-4514-b43d-5c462533947b-workload-socket\") pod \"097a37e6-fa47-4514-b43d-5c462533947b\" (UID: \"097a37e6-fa47-4514-b43d-5c462533947b\") "
Oct 30 19:44:34.662574 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:34.662524    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/097a37e6-fa47-4514-b43d-5c462533947b-kube-api-access-l9mfw\" (UniqueName: \"kubernetes.io/projected/097a37e6-fa47-4514-b43d-5c462533947b-kube-api-access-l9mfw\") pod \"097a37e6-fa47-4514-b43d-5c462533947b\" (UID: \"097a37e6-fa47-4514-b43d-5c462533947b\") "
Oct 30 19:44:34.662574 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:34.662556    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/097a37e6-fa47-4514-b43d-5c462533947b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/097a37e6-fa47-4514-b43d-5c462533947b-istio-data\") pod \"097a37e6-fa47-4514-b43d-5c462533947b\" (UID: \"097a37e6-fa47-4514-b43d-5c462533947b\") "
Oct 30 19:44:34.662871 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:34.662846    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/097a37e6-fa47-4514-b43d-5c462533947b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/097a37e6-fa47-4514-b43d-5c462533947b-istio-envoy\") pod \"097a37e6-fa47-4514-b43d-5c462533947b\" (UID: \"097a37e6-fa47-4514-b43d-5c462533947b\") "
Oct 30 19:44:34.663001 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:34.662886    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/097a37e6-fa47-4514-b43d-5c462533947b-istio-token\" (UniqueName: \"kubernetes.io/projected/097a37e6-fa47-4514-b43d-5c462533947b-istio-token\") pod \"097a37e6-fa47-4514-b43d-5c462533947b\" (UID: \"097a37e6-fa47-4514-b43d-5c462533947b\") "
Oct 30 19:44:34.663001 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:34.662917    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/097a37e6-fa47-4514-b43d-5c462533947b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/097a37e6-fa47-4514-b43d-5c462533947b-istiod-ca-cert\") pod \"097a37e6-fa47-4514-b43d-5c462533947b\" (UID: \"097a37e6-fa47-4514-b43d-5c462533947b\") "
Oct 30 19:44:34.663001 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:34.662949    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/097a37e6-fa47-4514-b43d-5c462533947b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/097a37e6-fa47-4514-b43d-5c462533947b-credential-socket\") pod \"097a37e6-fa47-4514-b43d-5c462533947b\" (UID: \"097a37e6-fa47-4514-b43d-5c462533947b\") "
Oct 30 19:44:34.663001 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:34.662982    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/097a37e6-fa47-4514-b43d-5c462533947b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/097a37e6-fa47-4514-b43d-5c462533947b-workload-certs\") pod \"097a37e6-fa47-4514-b43d-5c462533947b\" (UID: \"097a37e6-fa47-4514-b43d-5c462533947b\") "
Oct 30 19:44:34.663584 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:34.663548    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/097a37e6-fa47-4514-b43d-5c462533947b-workload-certs" pod "097a37e6-fa47-4514-b43d-5c462533947b" (UID: "097a37e6-fa47-4514-b43d-5c462533947b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:34.663584 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:34.663564    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/097a37e6-fa47-4514-b43d-5c462533947b-istio-data" pod "097a37e6-fa47-4514-b43d-5c462533947b" (UID: "097a37e6-fa47-4514-b43d-5c462533947b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:34.663918 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:34.663892    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/097a37e6-fa47-4514-b43d-5c462533947b-istiod-ca-cert" pod "097a37e6-fa47-4514-b43d-5c462533947b" (UID: "097a37e6-fa47-4514-b43d-5c462533947b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:34.664017 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:34.663957    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/097a37e6-fa47-4514-b43d-5c462533947b-credential-socket" pod "097a37e6-fa47-4514-b43d-5c462533947b" (UID: "097a37e6-fa47-4514-b43d-5c462533947b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:34.664585 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:34.664562    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/097a37e6-fa47-4514-b43d-5c462533947b-workload-socket" pod "097a37e6-fa47-4514-b43d-5c462533947b" (UID: "097a37e6-fa47-4514-b43d-5c462533947b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:34.665402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:34.665380    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/097a37e6-fa47-4514-b43d-5c462533947b-istio-podinfo" pod "097a37e6-fa47-4514-b43d-5c462533947b" (UID: "097a37e6-fa47-4514-b43d-5c462533947b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:44:34.665779 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:34.665755    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/097a37e6-fa47-4514-b43d-5c462533947b-istio-envoy" pod "097a37e6-fa47-4514-b43d-5c462533947b" (UID: "097a37e6-fa47-4514-b43d-5c462533947b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:34.666476 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:34.666457    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/097a37e6-fa47-4514-b43d-5c462533947b-istio-token" pod "097a37e6-fa47-4514-b43d-5c462533947b" (UID: "097a37e6-fa47-4514-b43d-5c462533947b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:34.666605 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:34.666589    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/097a37e6-fa47-4514-b43d-5c462533947b-kube-api-access-l9mfw" pod "097a37e6-fa47-4514-b43d-5c462533947b" (UID: "097a37e6-fa47-4514-b43d-5c462533947b"). InnerVolumeSpecName "kube-api-access-l9mfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:34.764313 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:34.764280    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/097a37e6-fa47-4514-b43d-5c462533947b-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:34.764313 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:34.764306    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/097a37e6-fa47-4514-b43d-5c462533947b-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:34.764313 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:34.764315    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/097a37e6-fa47-4514-b43d-5c462533947b-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:34.764605 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:34.764325    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9mfw\" (UniqueName: \"kubernetes.io/projected/097a37e6-fa47-4514-b43d-5c462533947b-kube-api-access-l9mfw\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:34.764605 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:34.764335    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/097a37e6-fa47-4514-b43d-5c462533947b-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:34.764605 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:34.764343    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/097a37e6-fa47-4514-b43d-5c462533947b-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:34.764605 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:34.764351    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/097a37e6-fa47-4514-b43d-5c462533947b-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:34.764605 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:34.764360    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/097a37e6-fa47-4514-b43d-5c462533947b-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:34.764605 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:34.764368    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/097a37e6-fa47-4514-b43d-5c462533947b-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:35.562544 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:35.562511    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj"
Oct 30 19:44:35.562544 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:35.562540    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj" event={"ID":"097a37e6-fa47-4514-b43d-5c462533947b","Type":"ContainerDied","Data":"0882398f55ff8bb84f2a51ff5801e6da8ad5cfb35c03ed55a627d8cfc8a343a1"}
Oct 30 19:44:35.563278 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:35.562622    5043 scope.go:122] "RemoveContainer" containerID="0d8116adb542a69bb69ceaba30af99c1caedb9f5dacf472d117b787d3714c39b"
Oct 30 19:44:35.590567 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:35.590536    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj"]
Oct 30 19:44:35.594140 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:35.594115    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-89a3f6-openshift-default-547f79fc89-d75jj"]
Oct 30 19:44:36.650385 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:36.650354    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="097a37e6-fa47-4514-b43d-5c462533947b" path="/var/lib/kubelet/pods/097a37e6-fa47-4514-b43d-5c462533947b/volumes"
Oct 30 19:44:36.650839 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:36.650669    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a97ffc3b-2lkdh"]
Oct 30 19:44:36.651195 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:36.651182    5043 cpu_manager.go:414] "containerMap: removing container" podUID="097a37e6-fa47-4514-b43d-5c462533947b" containerName="istio-proxy"
Oct 30 19:44:36.651247 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:36.651196    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="097a37e6-fa47-4514-b43d-5c462533947b" containerName="istio-proxy"
Oct 30 19:44:36.651247 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:36.651216    5043 cpu_manager.go:414] "containerMap: removing container" podUID="1ed7ff0c-6ec2-443f-b1d8-8f7cbf8238ce" containerName="payload-processing"
Oct 30 19:44:36.651247 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:36.651221    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ed7ff0c-6ec2-443f-b1d8-8f7cbf8238ce" containerName="payload-processing"
Oct 30 19:44:36.651247 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:36.651232    5043 cpu_manager.go:414] "containerMap: removing container" podUID="f791cec1-a1e4-4067-9221-cc04dafba0eb" containerName="istio-proxy"
Oct 30 19:44:36.651247 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:36.651236    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f791cec1-a1e4-4067-9221-cc04dafba0eb" containerName="istio-proxy"
Oct 30 19:44:36.651247 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:36.651248    5043 cpu_manager.go:414] "containerMap: removing container" podUID="4ef41111-49f3-4fd9-9587-2a1934f50797" containerName="payload-pre-processing"
Oct 30 19:44:36.651460 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:36.651252    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ef41111-49f3-4fd9-9587-2a1934f50797" containerName="payload-pre-processing"
Oct 30 19:44:36.651460 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:36.651270    5043 cpu_manager.go:414] "containerMap: removing container" podUID="f791cec1-a1e4-4067-9221-cc04dafba0eb" containerName="istio-proxy"
Oct 30 19:44:36.651460 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:36.651274    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f791cec1-a1e4-4067-9221-cc04dafba0eb" containerName="istio-proxy"
Oct 30 19:44:36.651460 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:36.651282    5043 cpu_manager.go:414] "containerMap: removing container" podUID="edc81663-c247-4c5e-8d01-89aa8f804165" containerName="revoke-keys"
Oct 30 19:44:36.651460 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:36.651287    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edc81663-c247-4c5e-8d01-89aa8f804165" containerName="revoke-keys"
Oct 30 19:44:36.651460 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:36.651362    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="f791cec1-a1e4-4067-9221-cc04dafba0eb" containerName="istio-proxy"
Oct 30 19:44:36.651460 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:36.651369    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="edc81663-c247-4c5e-8d01-89aa8f804165" containerName="revoke-keys"
Oct 30 19:44:36.651460 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:36.651379    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="097a37e6-fa47-4514-b43d-5c462533947b" containerName="istio-proxy"
Oct 30 19:44:36.651460 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:36.651387    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ed7ff0c-6ec2-443f-b1d8-8f7cbf8238ce" containerName="payload-processing"
Oct 30 19:44:36.651460 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:36.651396    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ef41111-49f3-4fd9-9587-2a1934f50797" containerName="payload-pre-processing"
Oct 30 19:44:36.656307 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:36.656287    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97ffc3b-2lkdh"
Oct 30 19:44:36.665359 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:36.665331    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a97ffc3b-2lkdh"]
Oct 30 19:44:36.666030 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:36.666012    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:44:36.666200 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:36.666133    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:44:36.667305 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:36.667285    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:44:36.667305 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:36.667300    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:44:36.707374 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:36.707344    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f4e49b7-9e2c-4479-a189-92a079a19e30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a97ffc3b-2lkdh\" (UID: \"9f4e49b7-9e2c-4479-a189-92a079a19e30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97ffc3b-2lkdh"
Oct 30 19:44:36.707543 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:36.707415    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f4e49b7-9e2c-4479-a189-92a079a19e30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a97ffc3b-2lkdh\" (UID: \"9f4e49b7-9e2c-4479-a189-92a079a19e30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97ffc3b-2lkdh"
Oct 30 19:44:36.707543 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:36.707517    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j45p5\" (UniqueName: \"kubernetes.io/projected/9f4e49b7-9e2c-4479-a189-92a079a19e30-kube-api-access-j45p5\") pod \"maas-api-revoke-sub-a97ffc3b-2lkdh\" (UID: \"9f4e49b7-9e2c-4479-a189-92a079a19e30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97ffc3b-2lkdh"
Oct 30 19:44:36.794195 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:36.794153    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ac37c8-fqfbn"]
Oct 30 19:44:36.796101 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:36.796068    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ac37c8-fqfbn"]
Oct 30 19:44:36.808361 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:36.808328    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f4e49b7-9e2c-4479-a189-92a079a19e30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a97ffc3b-2lkdh\" (UID: \"9f4e49b7-9e2c-4479-a189-92a079a19e30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97ffc3b-2lkdh"
Oct 30 19:44:36.808497 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:36.808396    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j45p5\" (UniqueName: \"kubernetes.io/projected/9f4e49b7-9e2c-4479-a189-92a079a19e30-kube-api-access-j45p5\") pod \"maas-api-revoke-sub-a97ffc3b-2lkdh\" (UID: \"9f4e49b7-9e2c-4479-a189-92a079a19e30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97ffc3b-2lkdh"
Oct 30 19:44:36.808614 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:36.808595    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f4e49b7-9e2c-4479-a189-92a079a19e30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a97ffc3b-2lkdh\" (UID: \"9f4e49b7-9e2c-4479-a189-92a079a19e30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97ffc3b-2lkdh"
Oct 30 19:44:36.809051 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:36.809028    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f4e49b7-9e2c-4479-a189-92a079a19e30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a97ffc3b-2lkdh\" (UID: \"9f4e49b7-9e2c-4479-a189-92a079a19e30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97ffc3b-2lkdh"
Oct 30 19:44:36.811219 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:36.811198    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f4e49b7-9e2c-4479-a189-92a079a19e30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a97ffc3b-2lkdh\" (UID: \"9f4e49b7-9e2c-4479-a189-92a079a19e30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97ffc3b-2lkdh"
Oct 30 19:44:36.821568 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:36.821545    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j45p5\" (UniqueName: \"kubernetes.io/projected/9f4e49b7-9e2c-4479-a189-92a079a19e30-kube-api-access-j45p5\") pod \"maas-api-revoke-sub-a97ffc3b-2lkdh\" (UID: \"9f4e49b7-9e2c-4479-a189-92a079a19e30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97ffc3b-2lkdh"
Oct 30 19:44:36.978379 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:36.978348    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97ffc3b-2lkdh"
Oct 30 19:44:37.293690 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:37.293665    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a97ffc3b-2lkdh"]
Oct 30 19:44:37.296253 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:44:37.296219    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f4e49b7_9e2c_4479_a189_92a079a19e30.slice/crio-b4d776ffb8f252b306aa24d4ba21947a9800e009b2d384e995d4a1df7c89e70c WatchSource:0}: Error finding container b4d776ffb8f252b306aa24d4ba21947a9800e009b2d384e995d4a1df7c89e70c: Status 404 returned error can't find the container with id b4d776ffb8f252b306aa24d4ba21947a9800e009b2d384e995d4a1df7c89e70c
Oct 30 19:44:37.631804 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:37.631770    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97ffc3b-2lkdh" event={"ID":"9f4e49b7-9e2c-4479-a189-92a079a19e30","Type":"ContainerStarted","Data":"755b146060b518b204a514a2e189a1e2dbc126a51290640ddedc3203e9c55e4c"}
Oct 30 19:44:37.631940 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:37.631831    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97ffc3b-2lkdh" event={"ID":"9f4e49b7-9e2c-4479-a189-92a079a19e30","Type":"ContainerStarted","Data":"b4d776ffb8f252b306aa24d4ba21947a9800e009b2d384e995d4a1df7c89e70c"}
Oct 30 19:44:37.651228 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:37.651185    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97ffc3b-2lkdh" podStartSLOduration=1.65117236 podStartE2EDuration="1.65117236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:37.649177369 +0000 UTC m=+1697.160237334" watchObservedRunningTime="2026-10-30 19:44:37.65117236 +0000 UTC m=+1697.162232324"
Oct 30 19:44:38.328313 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:38.328227    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ac37c8-57cb47d8c-4cprd" podUID="42cfcd85-0bda-4e33-ae8c-f84eb90afd6b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:44:38.645077 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:38.644971    5043 generic.go:356] "Generic (PLEG): container finished" podID="9f4e49b7-9e2c-4479-a189-92a079a19e30" containerID="755b146060b518b204a514a2e189a1e2dbc126a51290640ddedc3203e9c55e4c" exitCode=0
Oct 30 19:44:38.646333 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:38.646301    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e417b0f-a2b8-4792-9733-76b3a97a33df" path="/var/lib/kubelet/pods/9e417b0f-a2b8-4792-9733-76b3a97a33df/volumes"
Oct 30 19:44:38.646700 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:38.646681    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97ffc3b-2lkdh" event={"ID":"9f4e49b7-9e2c-4479-a189-92a079a19e30","Type":"ContainerDied","Data":"755b146060b518b204a514a2e189a1e2dbc126a51290640ddedc3203e9c55e4c"}
Oct 30 19:44:39.492854 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:39.492815    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a544ff33-2sfhh"]
Oct 30 19:44:39.493542 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:39.493525    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="f791cec1-a1e4-4067-9221-cc04dafba0eb" containerName="istio-proxy"
Oct 30 19:44:39.497366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:39.497343    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a544ff33-2sfhh"
Oct 30 19:44:39.513019 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:39.512948    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a544ff33-2sfhh"]
Oct 30 19:44:39.518657 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:39.518629    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e142711-bpvzp"]
Oct 30 19:44:39.524533 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:39.524506    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e142711-bpvzp"
Oct 30 19:44:39.538982 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:39.538953    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e142711-bpvzp"]
Oct 30 19:44:39.557016 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:39.556983    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhwwx\" (UniqueName: \"kubernetes.io/projected/f5279675-574c-4cee-9a73-b4b18429b99d-kube-api-access-rhwwx\") pod \"maas-api-revoke-sub-3e142711-bpvzp\" (UID: \"f5279675-574c-4cee-9a73-b4b18429b99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e142711-bpvzp"
Oct 30 19:44:39.557159 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:39.557035    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtpbb\" (UniqueName: \"kubernetes.io/projected/358f00f6-d477-4a66-917d-d551097c6bcc-kube-api-access-wtpbb\") pod \"maas-api-revoke-sub-a544ff33-2sfhh\" (UID: \"358f00f6-d477-4a66-917d-d551097c6bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a544ff33-2sfhh"
Oct 30 19:44:39.557159 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:39.557110    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5279675-574c-4cee-9a73-b4b18429b99d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e142711-bpvzp\" (UID: \"f5279675-574c-4cee-9a73-b4b18429b99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e142711-bpvzp"
Oct 30 19:44:39.557253 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:39.557227    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/358f00f6-d477-4a66-917d-d551097c6bcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a544ff33-2sfhh\" (UID: \"358f00f6-d477-4a66-917d-d551097c6bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a544ff33-2sfhh"
Oct 30 19:44:39.557302 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:39.557288    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5279675-574c-4cee-9a73-b4b18429b99d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e142711-bpvzp\" (UID: \"f5279675-574c-4cee-9a73-b4b18429b99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e142711-bpvzp"
Oct 30 19:44:39.557602 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:39.557418    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/358f00f6-d477-4a66-917d-d551097c6bcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a544ff33-2sfhh\" (UID: \"358f00f6-d477-4a66-917d-d551097c6bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a544ff33-2sfhh"
Oct 30 19:44:39.656371 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:39.656330    5043 generic.go:356] "Generic (PLEG): container finished" podID="62c3de5f-9601-43f4-a89a-85c623e1222b" containerID="7ad16887948a82336cccb54ea5544800808823a9636386c02057bbaa8ac5316f" exitCode=0
Oct 30 19:44:39.656548 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:39.656400    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89a3f6-db46ffd87-8l4s5" event={"ID":"62c3de5f-9601-43f4-a89a-85c623e1222b","Type":"ContainerDied","Data":"7ad16887948a82336cccb54ea5544800808823a9636386c02057bbaa8ac5316f"}
Oct 30 19:44:39.658819 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:39.658789    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/358f00f6-d477-4a66-917d-d551097c6bcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a544ff33-2sfhh\" (UID: \"358f00f6-d477-4a66-917d-d551097c6bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a544ff33-2sfhh"
Oct 30 19:44:39.658997 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:39.658975    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhwwx\" (UniqueName: \"kubernetes.io/projected/f5279675-574c-4cee-9a73-b4b18429b99d-kube-api-access-rhwwx\") pod \"maas-api-revoke-sub-3e142711-bpvzp\" (UID: \"f5279675-574c-4cee-9a73-b4b18429b99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e142711-bpvzp"
Oct 30 19:44:39.659070 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:39.659019    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtpbb\" (UniqueName: \"kubernetes.io/projected/358f00f6-d477-4a66-917d-d551097c6bcc-kube-api-access-wtpbb\") pod \"maas-api-revoke-sub-a544ff33-2sfhh\" (UID: \"358f00f6-d477-4a66-917d-d551097c6bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a544ff33-2sfhh"
Oct 30 19:44:39.659070 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:39.659055    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5279675-574c-4cee-9a73-b4b18429b99d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e142711-bpvzp\" (UID: \"f5279675-574c-4cee-9a73-b4b18429b99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e142711-bpvzp"
Oct 30 19:44:39.659724 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:39.659098    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/358f00f6-d477-4a66-917d-d551097c6bcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a544ff33-2sfhh\" (UID: \"358f00f6-d477-4a66-917d-d551097c6bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a544ff33-2sfhh"
Oct 30 19:44:39.660383 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:39.660354    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5279675-574c-4cee-9a73-b4b18429b99d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e142711-bpvzp\" (UID: \"f5279675-574c-4cee-9a73-b4b18429b99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e142711-bpvzp"
Oct 30 19:44:39.660564 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:39.660539    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/358f00f6-d477-4a66-917d-d551097c6bcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a544ff33-2sfhh\" (UID: \"358f00f6-d477-4a66-917d-d551097c6bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a544ff33-2sfhh"
Oct 30 19:44:39.660659 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:39.660639    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5279675-574c-4cee-9a73-b4b18429b99d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e142711-bpvzp\" (UID: \"f5279675-574c-4cee-9a73-b4b18429b99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e142711-bpvzp"
Oct 30 19:44:39.671110 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:44:39.670337    5043 logging.go:55] [core] [Channel #3897 SubChannel #3898]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.196:50052", ServerName: "10.132.0.196:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.196:50052: connect: connection refused"
Oct 30 19:44:39.671110 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:39.670477    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5279675-574c-4cee-9a73-b4b18429b99d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e142711-bpvzp\" (UID: \"f5279675-574c-4cee-9a73-b4b18429b99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e142711-bpvzp"
Oct 30 19:44:39.671110 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:39.670790    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/358f00f6-d477-4a66-917d-d551097c6bcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a544ff33-2sfhh\" (UID: \"358f00f6-d477-4a66-917d-d551097c6bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a544ff33-2sfhh"
Oct 30 19:44:39.675786 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:39.675714    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtpbb\" (UniqueName: \"kubernetes.io/projected/358f00f6-d477-4a66-917d-d551097c6bcc-kube-api-access-wtpbb\") pod \"maas-api-revoke-sub-a544ff33-2sfhh\" (UID: \"358f00f6-d477-4a66-917d-d551097c6bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a544ff33-2sfhh"
Oct 30 19:44:39.678657 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:39.678613    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhwwx\" (UniqueName: \"kubernetes.io/projected/f5279675-574c-4cee-9a73-b4b18429b99d-kube-api-access-rhwwx\") pod \"maas-api-revoke-sub-3e142711-bpvzp\" (UID: \"f5279675-574c-4cee-9a73-b4b18429b99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e142711-bpvzp"
Oct 30 19:44:39.813962 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:39.813920    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89a3f6-db46ffd87-8l4s5"
Oct 30 19:44:39.871783 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:39.864246    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62c3de5f-9601-43f4-a89a-85c623e1222b-kube-api-access-869j4\" (UniqueName: \"kubernetes.io/projected/62c3de5f-9601-43f4-a89a-85c623e1222b-kube-api-access-869j4\") pod \"62c3de5f-9601-43f4-a89a-85c623e1222b\" (UID: \"62c3de5f-9601-43f4-a89a-85c623e1222b\") "
Oct 30 19:44:39.871783 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:39.864348    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62c3de5f-9601-43f4-a89a-85c623e1222b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62c3de5f-9601-43f4-a89a-85c623e1222b-plugins-config-volume\") pod \"62c3de5f-9601-43f4-a89a-85c623e1222b\" (UID: \"62c3de5f-9601-43f4-a89a-85c623e1222b\") "
Oct 30 19:44:39.871783 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:39.865238    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62c3de5f-9601-43f4-a89a-85c623e1222b-plugins-config-volume" pod "62c3de5f-9601-43f4-a89a-85c623e1222b" (UID: "62c3de5f-9601-43f4-a89a-85c623e1222b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:39.880004 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:39.879962    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c3de5f-9601-43f4-a89a-85c623e1222b-kube-api-access-869j4" pod "62c3de5f-9601-43f4-a89a-85c623e1222b" (UID: "62c3de5f-9601-43f4-a89a-85c623e1222b"). InnerVolumeSpecName "kube-api-access-869j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:39.882527 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:39.882496    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97ffc3b-2lkdh"
Oct 30 19:44:39.884409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:39.884190    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a544ff33-2sfhh"
Oct 30 19:44:39.892303 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:39.892275    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e142711-bpvzp"
Oct 30 19:44:39.966650 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:39.966335    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f4e49b7-9e2c-4479-a189-92a079a19e30-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f4e49b7-9e2c-4479-a189-92a079a19e30-maas-api-xks-ca\") pod \"9f4e49b7-9e2c-4479-a189-92a079a19e30\" (UID: \"9f4e49b7-9e2c-4479-a189-92a079a19e30\") "
Oct 30 19:44:39.966650 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:39.966615    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f4e49b7-9e2c-4479-a189-92a079a19e30-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f4e49b7-9e2c-4479-a189-92a079a19e30-maas-api-openshift-ca\") pod \"9f4e49b7-9e2c-4479-a189-92a079a19e30\" (UID: \"9f4e49b7-9e2c-4479-a189-92a079a19e30\") "
Oct 30 19:44:39.967690 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:39.967434    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f4e49b7-9e2c-4479-a189-92a079a19e30-kube-api-access-j45p5\" (UniqueName: \"kubernetes.io/projected/9f4e49b7-9e2c-4479-a189-92a079a19e30-kube-api-access-j45p5\") pod \"9f4e49b7-9e2c-4479-a189-92a079a19e30\" (UID: \"9f4e49b7-9e2c-4479-a189-92a079a19e30\") "
Oct 30 19:44:39.968653 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:39.968152    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-869j4\" (UniqueName: \"kubernetes.io/projected/62c3de5f-9601-43f4-a89a-85c623e1222b-kube-api-access-869j4\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:39.968653 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:39.968176    5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62c3de5f-9601-43f4-a89a-85c623e1222b-plugins-config-volume\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:39.970673 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:39.970635    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f4e49b7-9e2c-4479-a189-92a079a19e30-maas-api-openshift-ca" pod "9f4e49b7-9e2c-4479-a189-92a079a19e30" (UID: "9f4e49b7-9e2c-4479-a189-92a079a19e30"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:39.974649 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:39.974611    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f4e49b7-9e2c-4479-a189-92a079a19e30-kube-api-access-j45p5" pod "9f4e49b7-9e2c-4479-a189-92a079a19e30" (UID: "9f4e49b7-9e2c-4479-a189-92a079a19e30"). InnerVolumeSpecName "kube-api-access-j45p5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:39.976713 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:39.976677    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f4e49b7-9e2c-4479-a189-92a079a19e30-maas-api-xks-ca" pod "9f4e49b7-9e2c-4479-a189-92a079a19e30" (UID: "9f4e49b7-9e2c-4479-a189-92a079a19e30"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:44:40.069988 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:40.069906    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f4e49b7-9e2c-4479-a189-92a079a19e30-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:40.069988 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:40.069937    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j45p5\" (UniqueName: \"kubernetes.io/projected/9f4e49b7-9e2c-4479-a189-92a079a19e30-kube-api-access-j45p5\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:40.069988 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:40.069953    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f4e49b7-9e2c-4479-a189-92a079a19e30-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:40.207896 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:40.202987    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e142711-bpvzp"]
Oct 30 19:44:40.217608 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:40.213353    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a544ff33-2sfhh"]
Oct 30 19:44:40.677987 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:40.677940    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89a3f6-db46ffd87-8l4s5" podUID="62c3de5f-9601-43f4-a89a-85c623e1222b" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.196:50052\" within 1s: context deadline exceeded"
Oct 30 19:44:40.678448 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:40.678051    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89a3f6-db46ffd87-8l4s5"
Oct 30 19:44:40.678448 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:44:40.678111    5043 logging.go:55] [core] [Channel #3897 SubChannel #3898]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.196:50052", ServerName: "10.132.0.196:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.196:50052: operation was canceled"
Oct 30 19:44:40.680305 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:40.680285    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89a3f6-db46ffd87-8l4s5"
Oct 30 19:44:40.680616 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:40.680570    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89a3f6-db46ffd87-8l4s5" event={"ID":"62c3de5f-9601-43f4-a89a-85c623e1222b","Type":"ContainerDied","Data":"c55e1ce7e0548eba77d04bbe4d83caaaff86831cd936c8e4d1fe2182677814b6"}
Oct 30 19:44:40.680781 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:40.680661    5043 scope.go:122] "RemoveContainer" containerID="7ad16887948a82336cccb54ea5544800808823a9636386c02057bbaa8ac5316f"
Oct 30 19:44:40.683119 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:40.683089    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97ffc3b-2lkdh" event={"ID":"9f4e49b7-9e2c-4479-a189-92a079a19e30","Type":"ContainerDied","Data":"b4d776ffb8f252b306aa24d4ba21947a9800e009b2d384e995d4a1df7c89e70c"}
Oct 30 19:44:40.683209 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:40.683164    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4d776ffb8f252b306aa24d4ba21947a9800e009b2d384e995d4a1df7c89e70c"
Oct 30 19:44:40.683264 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:40.683254    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97ffc3b-2lkdh"
Oct 30 19:44:40.687435 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:40.687405    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e142711-bpvzp" event={"ID":"f5279675-574c-4cee-9a73-b4b18429b99d","Type":"ContainerStarted","Data":"e9da49ea17baec443c195e05040526f56bb191a47cfb302203d1039bd131dd63"}
Oct 30 19:44:40.691474 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:40.691444    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a544ff33-2sfhh" event={"ID":"358f00f6-d477-4a66-917d-d551097c6bcc","Type":"ContainerStarted","Data":"d41bcb7e34dcc5ee6c84016514cbe8032a6d2456504ecbde074c984af895bcac"}
Oct 30 19:44:40.710468 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:40.710441    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-89a3f6-db46ffd87-8l4s5"]
Oct 30 19:44:40.719651 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:40.719627    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-89a3f6-db46ffd87-8l4s5"]
Oct 30 19:44:41.703476 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:41.703381    5043 generic.go:356] "Generic (PLEG): container finished" podID="f5279675-574c-4cee-9a73-b4b18429b99d" containerID="37c1f7179985033b12dd2f89ba657862ea6a27b3972be1986ecb9f7439477f5c" exitCode=0
Oct 30 19:44:41.703983 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:41.703480    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e142711-bpvzp" event={"ID":"f5279675-574c-4cee-9a73-b4b18429b99d","Type":"ContainerDied","Data":"37c1f7179985033b12dd2f89ba657862ea6a27b3972be1986ecb9f7439477f5c"}
Oct 30 19:44:41.705564 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:41.705536    5043 generic.go:356] "Generic (PLEG): container finished" podID="358f00f6-d477-4a66-917d-d551097c6bcc" containerID="472ebf84359c01ca2d015a9fdcd914a1062e20dd8f483d6036989d602ebb9efd" exitCode=0
Oct 30 19:44:41.705711 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:41.705625    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a544ff33-2sfhh" event={"ID":"358f00f6-d477-4a66-917d-d551097c6bcc","Type":"ContainerDied","Data":"472ebf84359c01ca2d015a9fdcd914a1062e20dd8f483d6036989d602ebb9efd"}
Oct 30 19:44:42.646970 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:42.646931    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62c3de5f-9601-43f4-a89a-85c623e1222b" path="/var/lib/kubelet/pods/62c3de5f-9601-43f4-a89a-85c623e1222b/volumes"
Oct 30 19:44:43.067856 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:43.067828    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e142711-bpvzp"
Oct 30 19:44:43.069883 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:43.069866    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a544ff33-2sfhh"
Oct 30 19:44:43.165765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:43.164048    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/358f00f6-d477-4a66-917d-d551097c6bcc-kube-api-access-wtpbb\" (UniqueName: \"kubernetes.io/projected/358f00f6-d477-4a66-917d-d551097c6bcc-kube-api-access-wtpbb\") pod \"358f00f6-d477-4a66-917d-d551097c6bcc\" (UID: \"358f00f6-d477-4a66-917d-d551097c6bcc\") "
Oct 30 19:44:43.165765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:43.164107    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/358f00f6-d477-4a66-917d-d551097c6bcc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/358f00f6-d477-4a66-917d-d551097c6bcc-maas-api-openshift-ca\") pod \"358f00f6-d477-4a66-917d-d551097c6bcc\" (UID: \"358f00f6-d477-4a66-917d-d551097c6bcc\") "
Oct 30 19:44:43.165765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:43.164147    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5279675-574c-4cee-9a73-b4b18429b99d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5279675-574c-4cee-9a73-b4b18429b99d-maas-api-xks-ca\") pod \"f5279675-574c-4cee-9a73-b4b18429b99d\" (UID: \"f5279675-574c-4cee-9a73-b4b18429b99d\") "
Oct 30 19:44:43.165765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:43.164244    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5279675-574c-4cee-9a73-b4b18429b99d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5279675-574c-4cee-9a73-b4b18429b99d-maas-api-openshift-ca\") pod \"f5279675-574c-4cee-9a73-b4b18429b99d\" (UID: \"f5279675-574c-4cee-9a73-b4b18429b99d\") "
Oct 30 19:44:43.165765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:43.164318    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5279675-574c-4cee-9a73-b4b18429b99d-kube-api-access-rhwwx\" (UniqueName: \"kubernetes.io/projected/f5279675-574c-4cee-9a73-b4b18429b99d-kube-api-access-rhwwx\") pod \"f5279675-574c-4cee-9a73-b4b18429b99d\" (UID: \"f5279675-574c-4cee-9a73-b4b18429b99d\") "
Oct 30 19:44:43.165765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:43.164402    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/358f00f6-d477-4a66-917d-d551097c6bcc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/358f00f6-d477-4a66-917d-d551097c6bcc-maas-api-xks-ca\") pod \"358f00f6-d477-4a66-917d-d551097c6bcc\" (UID: \"358f00f6-d477-4a66-917d-d551097c6bcc\") "
Oct 30 19:44:43.169761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:43.166289    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/358f00f6-d477-4a66-917d-d551097c6bcc-maas-api-openshift-ca" pod "358f00f6-d477-4a66-917d-d551097c6bcc" (UID: "358f00f6-d477-4a66-917d-d551097c6bcc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:43.169761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:43.166691    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5279675-574c-4cee-9a73-b4b18429b99d-maas-api-openshift-ca" pod "f5279675-574c-4cee-9a73-b4b18429b99d" (UID: "f5279675-574c-4cee-9a73-b4b18429b99d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:43.180227 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:43.180184    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/358f00f6-d477-4a66-917d-d551097c6bcc-kube-api-access-wtpbb" pod "358f00f6-d477-4a66-917d-d551097c6bcc" (UID: "358f00f6-d477-4a66-917d-d551097c6bcc"). InnerVolumeSpecName "kube-api-access-wtpbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:43.180227 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:43.180226    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5279675-574c-4cee-9a73-b4b18429b99d-kube-api-access-rhwwx" pod "f5279675-574c-4cee-9a73-b4b18429b99d" (UID: "f5279675-574c-4cee-9a73-b4b18429b99d"). InnerVolumeSpecName "kube-api-access-rhwwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:43.182200 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:43.182171    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5279675-574c-4cee-9a73-b4b18429b99d-maas-api-xks-ca" pod "f5279675-574c-4cee-9a73-b4b18429b99d" (UID: "f5279675-574c-4cee-9a73-b4b18429b99d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:44:43.184291 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:43.184267    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/358f00f6-d477-4a66-917d-d551097c6bcc-maas-api-xks-ca" pod "358f00f6-d477-4a66-917d-d551097c6bcc" (UID: "358f00f6-d477-4a66-917d-d551097c6bcc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:44:43.266486 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:43.266441    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/358f00f6-d477-4a66-917d-d551097c6bcc-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:43.266486 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:43.266486    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5279675-574c-4cee-9a73-b4b18429b99d-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:43.266705 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:43.266505    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5279675-574c-4cee-9a73-b4b18429b99d-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:43.266705 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:43.266518    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhwwx\" (UniqueName: \"kubernetes.io/projected/f5279675-574c-4cee-9a73-b4b18429b99d-kube-api-access-rhwwx\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:43.266705 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:43.266532    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/358f00f6-d477-4a66-917d-d551097c6bcc-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:43.266705 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:43.266545    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtpbb\" (UniqueName: \"kubernetes.io/projected/358f00f6-d477-4a66-917d-d551097c6bcc-kube-api-access-wtpbb\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:43.727249 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:43.727224    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e142711-bpvzp"
Oct 30 19:44:43.727481 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:43.727269    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e142711-bpvzp" event={"ID":"f5279675-574c-4cee-9a73-b4b18429b99d","Type":"ContainerDied","Data":"e9da49ea17baec443c195e05040526f56bb191a47cfb302203d1039bd131dd63"}
Oct 30 19:44:43.727481 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:43.727331    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9da49ea17baec443c195e05040526f56bb191a47cfb302203d1039bd131dd63"
Oct 30 19:44:43.729041 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:43.729019    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a544ff33-2sfhh"
Oct 30 19:44:43.729041 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:43.729031    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a544ff33-2sfhh" event={"ID":"358f00f6-d477-4a66-917d-d551097c6bcc","Type":"ContainerDied","Data":"d41bcb7e34dcc5ee6c84016514cbe8032a6d2456504ecbde074c984af895bcac"}
Oct 30 19:44:43.729198 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:43.729073    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d41bcb7e34dcc5ee6c84016514cbe8032a6d2456504ecbde074c984af895bcac"
Oct 30 19:44:44.177643 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:44.177556    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d651c4-ww6bm"]
Oct 30 19:44:44.188923 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:44.188898    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d651c4-ww6bm"]
Oct 30 19:44:44.646960 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:44.646928    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edc81663-c247-4c5e-8d01-89aa8f804165" path="/var/lib/kubelet/pods/edc81663-c247-4c5e-8d01-89aa8f804165/volumes"
Oct 30 19:44:46.095959 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.095914    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main" containerID="cri-o://7996187c5b2d2513fb75dc201cc1aebb27aa9cf8db792fc0839d36a01ca852b7" gracePeriod=45
Oct 30 19:44:46.142465 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.142283    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" containerID="cri-o://5a146da3d9f677e3ecaeeb3ee4266570c623883eba84bc3ae2904db3849e85ba" gracePeriod=45
Oct 30 19:44:46.161700 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.161660    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": dial tcp 10.132.0.62:8000: connect: connection refused"
Oct 30 19:44:46.470818 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.470726    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"
Oct 30 19:44:46.473971 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.473949    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:44:46.623731 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.623650    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71b8006f-09c2-4f6a-a315-3687f72aed60-home\" (UniqueName: \"kubernetes.io/empty-dir/71b8006f-09c2-4f6a-a315-3687f72aed60-home\") pod \"71b8006f-09c2-4f6a-a315-3687f72aed60\" (UID: \"71b8006f-09c2-4f6a-a315-3687f72aed60\") "
Oct 30 19:44:46.623731 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.623700    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e62bd6c5-e6ef-427d-a5de-4e2349026a2b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e62bd6c5-e6ef-427d-a5de-4e2349026a2b-tmp-dir\") pod \"e62bd6c5-e6ef-427d-a5de-4e2349026a2b\" (UID: \"e62bd6c5-e6ef-427d-a5de-4e2349026a2b\") "
Oct 30 19:44:46.623731 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.623751    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71b8006f-09c2-4f6a-a315-3687f72aed60-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71b8006f-09c2-4f6a-a315-3687f72aed60-model-cache\") pod \"71b8006f-09c2-4f6a-a315-3687f72aed60\" (UID: \"71b8006f-09c2-4f6a-a315-3687f72aed60\") "
Oct 30 19:44:46.624445 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.623769    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e62bd6c5-e6ef-427d-a5de-4e2349026a2b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e62bd6c5-e6ef-427d-a5de-4e2349026a2b-dshm\") pod \"e62bd6c5-e6ef-427d-a5de-4e2349026a2b\" (UID: \"e62bd6c5-e6ef-427d-a5de-4e2349026a2b\") "
Oct 30 19:44:46.624445 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.623803    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e62bd6c5-e6ef-427d-a5de-4e2349026a2b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e62bd6c5-e6ef-427d-a5de-4e2349026a2b-model-cache\") pod \"e62bd6c5-e6ef-427d-a5de-4e2349026a2b\" (UID: \"e62bd6c5-e6ef-427d-a5de-4e2349026a2b\") "
Oct 30 19:44:46.624445 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.623833    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71b8006f-09c2-4f6a-a315-3687f72aed60-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71b8006f-09c2-4f6a-a315-3687f72aed60-tmp-dir\") pod \"71b8006f-09c2-4f6a-a315-3687f72aed60\" (UID: \"71b8006f-09c2-4f6a-a315-3687f72aed60\") "
Oct 30 19:44:46.624445 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.623886    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e62bd6c5-e6ef-427d-a5de-4e2349026a2b-home\" (UniqueName: \"kubernetes.io/empty-dir/e62bd6c5-e6ef-427d-a5de-4e2349026a2b-home\") pod \"e62bd6c5-e6ef-427d-a5de-4e2349026a2b\" (UID: \"e62bd6c5-e6ef-427d-a5de-4e2349026a2b\") "
Oct 30 19:44:46.624445 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.623992    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71b8006f-09c2-4f6a-a315-3687f72aed60-dshm\" (UniqueName: \"kubernetes.io/empty-dir/71b8006f-09c2-4f6a-a315-3687f72aed60-dshm\") pod \"71b8006f-09c2-4f6a-a315-3687f72aed60\" (UID: \"71b8006f-09c2-4f6a-a315-3687f72aed60\") "
Oct 30 19:44:46.624445 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.623993    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b8006f-09c2-4f6a-a315-3687f72aed60-home" pod "71b8006f-09c2-4f6a-a315-3687f72aed60" (UID: "71b8006f-09c2-4f6a-a315-3687f72aed60"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:46.624445 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.624033    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b8006f-09c2-4f6a-a315-3687f72aed60-model-cache" pod "71b8006f-09c2-4f6a-a315-3687f72aed60" (UID: "71b8006f-09c2-4f6a-a315-3687f72aed60"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:46.624445 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.624056    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e62bd6c5-e6ef-427d-a5de-4e2349026a2b-tmp-dir" pod "e62bd6c5-e6ef-427d-a5de-4e2349026a2b" (UID: "e62bd6c5-e6ef-427d-a5de-4e2349026a2b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:46.624445 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.624097    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71b8006f-09c2-4f6a-a315-3687f72aed60-tls-certs\" (UniqueName: \"kubernetes.io/secret/71b8006f-09c2-4f6a-a315-3687f72aed60-tls-certs\") pod \"71b8006f-09c2-4f6a-a315-3687f72aed60\" (UID: \"71b8006f-09c2-4f6a-a315-3687f72aed60\") "
Oct 30 19:44:46.624445 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.624173    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e62bd6c5-e6ef-427d-a5de-4e2349026a2b-kube-api-access-68pkt\" (UniqueName: \"kubernetes.io/projected/e62bd6c5-e6ef-427d-a5de-4e2349026a2b-kube-api-access-68pkt\") pod \"e62bd6c5-e6ef-427d-a5de-4e2349026a2b\" (UID: \"e62bd6c5-e6ef-427d-a5de-4e2349026a2b\") "
Oct 30 19:44:46.624445 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.624183    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b8006f-09c2-4f6a-a315-3687f72aed60-tmp-dir" pod "71b8006f-09c2-4f6a-a315-3687f72aed60" (UID: "71b8006f-09c2-4f6a-a315-3687f72aed60"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:46.624445 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.624203    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e62bd6c5-e6ef-427d-a5de-4e2349026a2b-home" pod "e62bd6c5-e6ef-427d-a5de-4e2349026a2b" (UID: "e62bd6c5-e6ef-427d-a5de-4e2349026a2b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:46.624445 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.624283    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e62bd6c5-e6ef-427d-a5de-4e2349026a2b-tls-certs\" (UniqueName: \"kubernetes.io/secret/e62bd6c5-e6ef-427d-a5de-4e2349026a2b-tls-certs\") pod \"e62bd6c5-e6ef-427d-a5de-4e2349026a2b\" (UID: \"e62bd6c5-e6ef-427d-a5de-4e2349026a2b\") "
Oct 30 19:44:46.624445 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.624333    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71b8006f-09c2-4f6a-a315-3687f72aed60-kube-api-access-vrv2t\" (UniqueName: \"kubernetes.io/projected/71b8006f-09c2-4f6a-a315-3687f72aed60-kube-api-access-vrv2t\") pod \"71b8006f-09c2-4f6a-a315-3687f72aed60\" (UID: \"71b8006f-09c2-4f6a-a315-3687f72aed60\") "
Oct 30 19:44:46.624445 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.624151    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e62bd6c5-e6ef-427d-a5de-4e2349026a2b-model-cache" pod "e62bd6c5-e6ef-427d-a5de-4e2349026a2b" (UID: "e62bd6c5-e6ef-427d-a5de-4e2349026a2b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:46.625835 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.625132    5043 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e62bd6c5-e6ef-427d-a5de-4e2349026a2b-tmp-dir\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:46.625835 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.625152    5043 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71b8006f-09c2-4f6a-a315-3687f72aed60-model-cache\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:46.625835 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.625166    5043 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e62bd6c5-e6ef-427d-a5de-4e2349026a2b-model-cache\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:46.625835 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.625180    5043 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71b8006f-09c2-4f6a-a315-3687f72aed60-tmp-dir\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:46.625835 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.625193    5043 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e62bd6c5-e6ef-427d-a5de-4e2349026a2b-home\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:46.625835 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.625206    5043 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71b8006f-09c2-4f6a-a315-3687f72aed60-home\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:46.628214 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.628123    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b8006f-09c2-4f6a-a315-3687f72aed60-dshm" pod "71b8006f-09c2-4f6a-a315-3687f72aed60" (UID: "71b8006f-09c2-4f6a-a315-3687f72aed60"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:46.628214 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.628137    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e62bd6c5-e6ef-427d-a5de-4e2349026a2b-tls-certs" pod "e62bd6c5-e6ef-427d-a5de-4e2349026a2b" (UID: "e62bd6c5-e6ef-427d-a5de-4e2349026a2b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:44:46.628891 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.628806    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e62bd6c5-e6ef-427d-a5de-4e2349026a2b-kube-api-access-68pkt" pod "e62bd6c5-e6ef-427d-a5de-4e2349026a2b" (UID: "e62bd6c5-e6ef-427d-a5de-4e2349026a2b"). InnerVolumeSpecName "kube-api-access-68pkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:46.629419 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.629344    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71b8006f-09c2-4f6a-a315-3687f72aed60-kube-api-access-vrv2t" pod "71b8006f-09c2-4f6a-a315-3687f72aed60" (UID: "71b8006f-09c2-4f6a-a315-3687f72aed60"). InnerVolumeSpecName "kube-api-access-vrv2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:46.630250 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.630187    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e62bd6c5-e6ef-427d-a5de-4e2349026a2b-dshm" pod "e62bd6c5-e6ef-427d-a5de-4e2349026a2b" (UID: "e62bd6c5-e6ef-427d-a5de-4e2349026a2b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:46.632929 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.632805    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71b8006f-09c2-4f6a-a315-3687f72aed60-tls-certs" pod "71b8006f-09c2-4f6a-a315-3687f72aed60" (UID: "71b8006f-09c2-4f6a-a315-3687f72aed60"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:44:46.726312 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.726285    5043 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71b8006f-09c2-4f6a-a315-3687f72aed60-dshm\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:46.726312 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.726315    5043 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71b8006f-09c2-4f6a-a315-3687f72aed60-tls-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:46.726494 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.726329    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68pkt\" (UniqueName: \"kubernetes.io/projected/e62bd6c5-e6ef-427d-a5de-4e2349026a2b-kube-api-access-68pkt\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:46.726494 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.726342    5043 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e62bd6c5-e6ef-427d-a5de-4e2349026a2b-tls-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:46.726494 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.726354    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrv2t\" (UniqueName: \"kubernetes.io/projected/71b8006f-09c2-4f6a-a315-3687f72aed60-kube-api-access-vrv2t\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:46.726494 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.726366    5043 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e62bd6c5-e6ef-427d-a5de-4e2349026a2b-dshm\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:46.753390 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.753358    5043 generic.go:356] "Generic (PLEG): container finished" podID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerID="7996187c5b2d2513fb75dc201cc1aebb27aa9cf8db792fc0839d36a01ca852b7" exitCode=0
Oct 30 19:44:46.753539 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.753401    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"
Oct 30 19:44:46.753539 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.753438    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" event={"ID":"71b8006f-09c2-4f6a-a315-3687f72aed60","Type":"ContainerDied","Data":"7996187c5b2d2513fb75dc201cc1aebb27aa9cf8db792fc0839d36a01ca852b7"}
Oct 30 19:44:46.753539 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.753482    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5" event={"ID":"71b8006f-09c2-4f6a-a315-3687f72aed60","Type":"ContainerDied","Data":"58b4ef32b28df501b45992c8bef25c08bb984c599bdcc4ed463d039bfdd9d71c"}
Oct 30 19:44:46.753539 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.753502    5043 scope.go:122] "RemoveContainer" containerID="7996187c5b2d2513fb75dc201cc1aebb27aa9cf8db792fc0839d36a01ca852b7"
Oct 30 19:44:46.755112 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.755072    5043 generic.go:356] "Generic (PLEG): container finished" podID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerID="5a146da3d9f677e3ecaeeb3ee4266570c623883eba84bc3ae2904db3849e85ba" exitCode=0
Oct 30 19:44:46.755237 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.755134    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"
Oct 30 19:44:46.759769 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.755985    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" event={"ID":"e62bd6c5-e6ef-427d-a5de-4e2349026a2b","Type":"ContainerDied","Data":"5a146da3d9f677e3ecaeeb3ee4266570c623883eba84bc3ae2904db3849e85ba"}
Oct 30 19:44:46.759769 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.756030    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5" event={"ID":"e62bd6c5-e6ef-427d-a5de-4e2349026a2b","Type":"ContainerDied","Data":"7f1d30b01283a588c0906052f7d4699c39346cd6495cb7512024a9ddeda45fa0"}
Oct 30 19:44:46.765197 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.765178    5043 scope.go:122] "RemoveContainer" containerID="7996187c5b2d2513fb75dc201cc1aebb27aa9cf8db792fc0839d36a01ca852b7"
Oct 30 19:44:46.765486 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:44:46.765457    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7996187c5b2d2513fb75dc201cc1aebb27aa9cf8db792fc0839d36a01ca852b7\": container with ID starting with 7996187c5b2d2513fb75dc201cc1aebb27aa9cf8db792fc0839d36a01ca852b7 not found: ID does not exist" containerID="7996187c5b2d2513fb75dc201cc1aebb27aa9cf8db792fc0839d36a01ca852b7"
Oct 30 19:44:46.765573 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.765488    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7996187c5b2d2513fb75dc201cc1aebb27aa9cf8db792fc0839d36a01ca852b7"} err="failed to get container status \"7996187c5b2d2513fb75dc201cc1aebb27aa9cf8db792fc0839d36a01ca852b7\": rpc error: code = NotFound desc = could not find container \"7996187c5b2d2513fb75dc201cc1aebb27aa9cf8db792fc0839d36a01ca852b7\": container with ID starting with 7996187c5b2d2513fb75dc201cc1aebb27aa9cf8db792fc0839d36a01ca852b7 not found: ID does not exist"
Oct 30 19:44:46.765573 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.765510    5043 scope.go:122] "RemoveContainer" containerID="5a146da3d9f677e3ecaeeb3ee4266570c623883eba84bc3ae2904db3849e85ba"
Oct 30 19:44:46.776955 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.776923    5043 scope.go:122] "RemoveContainer" containerID="5a146da3d9f677e3ecaeeb3ee4266570c623883eba84bc3ae2904db3849e85ba"
Oct 30 19:44:46.777271 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:44:46.777248    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a146da3d9f677e3ecaeeb3ee4266570c623883eba84bc3ae2904db3849e85ba\": container with ID starting with 5a146da3d9f677e3ecaeeb3ee4266570c623883eba84bc3ae2904db3849e85ba not found: ID does not exist" containerID="5a146da3d9f677e3ecaeeb3ee4266570c623883eba84bc3ae2904db3849e85ba"
Oct 30 19:44:46.777336 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.777282    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a146da3d9f677e3ecaeeb3ee4266570c623883eba84bc3ae2904db3849e85ba"} err="failed to get container status \"5a146da3d9f677e3ecaeeb3ee4266570c623883eba84bc3ae2904db3849e85ba\": rpc error: code = NotFound desc = could not find container \"5a146da3d9f677e3ecaeeb3ee4266570c623883eba84bc3ae2904db3849e85ba\": container with ID starting with 5a146da3d9f677e3ecaeeb3ee4266570c623883eba84bc3ae2904db3849e85ba not found: ID does not exist"
Oct 30 19:44:46.791606 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.791583    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"]
Oct 30 19:44:46.799015 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.798996    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-01b24e/distinct-2-w6-01b24e-kserve-578dd8fdd8-24fh5"]
Oct 30 19:44:46.814329 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.814310    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"]
Oct 30 19:44:46.817508 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:46.817491    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-01b24e/simulator-w6-01b24e-kserve-68b7c5d6d5-dfbn5"]
Oct 30 19:44:47.944992 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:47.944965    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaae805a-m4l2s"]
Oct 30 19:44:47.945551 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:47.945531    5043 cpu_manager.go:414] "containerMap: removing container" podUID="62c3de5f-9601-43f4-a89a-85c623e1222b" containerName="payload-pre-processing"
Oct 30 19:44:47.945551 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:47.945548    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62c3de5f-9601-43f4-a89a-85c623e1222b" containerName="payload-pre-processing"
Oct 30 19:44:47.945705 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:47.945572    5043 cpu_manager.go:414] "containerMap: removing container" podUID="358f00f6-d477-4a66-917d-d551097c6bcc" containerName="revoke-keys"
Oct 30 19:44:47.945705 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:47.945579    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="358f00f6-d477-4a66-917d-d551097c6bcc" containerName="revoke-keys"
Oct 30 19:44:47.945705 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:47.945585    5043 cpu_manager.go:414] "containerMap: removing container" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main"
Oct 30 19:44:47.945705 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:47.945591    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main"
Oct 30 19:44:47.945705 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:47.945614    5043 cpu_manager.go:414] "containerMap: removing container" podUID="f5279675-574c-4cee-9a73-b4b18429b99d" containerName="revoke-keys"
Oct 30 19:44:47.945705 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:47.945619    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5279675-574c-4cee-9a73-b4b18429b99d" containerName="revoke-keys"
Oct 30 19:44:47.945705 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:47.945631    5043 cpu_manager.go:414] "containerMap: removing container" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main"
Oct 30 19:44:47.945705 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:47.945635    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main"
Oct 30 19:44:47.945705 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:47.945649    5043 cpu_manager.go:414] "containerMap: removing container" podUID="9f4e49b7-9e2c-4479-a189-92a079a19e30" containerName="revoke-keys"
Oct 30 19:44:47.945705 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:47.945653    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f4e49b7-9e2c-4479-a189-92a079a19e30" containerName="revoke-keys"
Oct 30 19:44:47.946048 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:47.945769    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" containerName="main"
Oct 30 19:44:47.946048 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:47.945784    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f4e49b7-9e2c-4479-a189-92a079a19e30" containerName="revoke-keys"
Oct 30 19:44:47.946048 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:47.945792    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="62c3de5f-9601-43f4-a89a-85c623e1222b" containerName="payload-pre-processing"
Oct 30 19:44:47.946048 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:47.945799    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="358f00f6-d477-4a66-917d-d551097c6bcc" containerName="revoke-keys"
Oct 30 19:44:47.946048 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:47.945804    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" containerName="main"
Oct 30 19:44:47.946048 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:47.945812    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5279675-574c-4cee-9a73-b4b18429b99d" containerName="revoke-keys"
Oct 30 19:44:47.950790 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:47.950772    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaae805a-m4l2s"
Oct 30 19:44:47.955973 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:47.955945    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:44:47.956111 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:47.956022    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:44:47.956280 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:47.956144    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:44:47.956280 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:47.956149    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:44:47.961420 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:47.961398    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaae805a-m4l2s"]
Oct 30 19:44:48.099600 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:48.099570    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r"]
Oct 30 19:44:48.099822 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:48.099798    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r" podUID="76b4e8e9-54ac-4461-b99c-a6f0361505b6" containerName="istio-proxy" containerID="cri-o://8308f4a0b3eb68229f823b999921178a37ea37806d32fd6f180b18d02137155e" gracePeriod=30
Oct 30 19:44:48.143961 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:48.143932    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8lrz\" (UniqueName: \"kubernetes.io/projected/14a645f0-5289-46a9-ab64-2743663bb5d0-kube-api-access-s8lrz\") pod \"maas-api-revoke-sub-eaae805a-m4l2s\" (UID: \"14a645f0-5289-46a9-ab64-2743663bb5d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaae805a-m4l2s"
Oct 30 19:44:48.144189 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:48.144169    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14a645f0-5289-46a9-ab64-2743663bb5d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eaae805a-m4l2s\" (UID: \"14a645f0-5289-46a9-ab64-2743663bb5d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaae805a-m4l2s"
Oct 30 19:44:48.144263 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:48.144221    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14a645f0-5289-46a9-ab64-2743663bb5d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eaae805a-m4l2s\" (UID: \"14a645f0-5289-46a9-ab64-2743663bb5d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaae805a-m4l2s"
Oct 30 19:44:48.250567 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:48.250528    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14a645f0-5289-46a9-ab64-2743663bb5d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eaae805a-m4l2s\" (UID: \"14a645f0-5289-46a9-ab64-2743663bb5d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaae805a-m4l2s"
Oct 30 19:44:48.250921 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:48.250896    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14a645f0-5289-46a9-ab64-2743663bb5d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eaae805a-m4l2s\" (UID: \"14a645f0-5289-46a9-ab64-2743663bb5d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaae805a-m4l2s"
Oct 30 19:44:48.251129 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:48.251109    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8lrz\" (UniqueName: \"kubernetes.io/projected/14a645f0-5289-46a9-ab64-2743663bb5d0-kube-api-access-s8lrz\") pod \"maas-api-revoke-sub-eaae805a-m4l2s\" (UID: \"14a645f0-5289-46a9-ab64-2743663bb5d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaae805a-m4l2s"
Oct 30 19:44:48.252227 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:48.252204    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14a645f0-5289-46a9-ab64-2743663bb5d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eaae805a-m4l2s\" (UID: \"14a645f0-5289-46a9-ab64-2743663bb5d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaae805a-m4l2s"
Oct 30 19:44:48.268537 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:48.268503    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14a645f0-5289-46a9-ab64-2743663bb5d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eaae805a-m4l2s\" (UID: \"14a645f0-5289-46a9-ab64-2743663bb5d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaae805a-m4l2s"
Oct 30 19:44:48.269082 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:48.269050    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8lrz\" (UniqueName: \"kubernetes.io/projected/14a645f0-5289-46a9-ab64-2743663bb5d0-kube-api-access-s8lrz\") pod \"maas-api-revoke-sub-eaae805a-m4l2s\" (UID: \"14a645f0-5289-46a9-ab64-2743663bb5d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaae805a-m4l2s"
Oct 30 19:44:48.328431 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:48.328391    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ac37c8-57cb47d8c-4cprd" podUID="42cfcd85-0bda-4e33-ae8c-f84eb90afd6b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:44:48.567300 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:48.567206    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaae805a-m4l2s"
Oct 30 19:44:48.650638 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:48.650600    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71b8006f-09c2-4f6a-a315-3687f72aed60" path="/var/lib/kubelet/pods/71b8006f-09c2-4f6a-a315-3687f72aed60/volumes"
Oct 30 19:44:48.651280 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:48.651258    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e62bd6c5-e6ef-427d-a5de-4e2349026a2b" path="/var/lib/kubelet/pods/e62bd6c5-e6ef-427d-a5de-4e2349026a2b/volumes"
Oct 30 19:44:48.727120 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:48.727090    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fdb355-pv6x5"]
Oct 30 19:44:48.732569 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:48.732552    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fdb355-pv6x5"
Oct 30 19:44:48.756445 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:48.756420    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fdb355-pv6x5"]
Oct 30 19:44:48.756786 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:48.756759    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29414e40-8647-43d5-972c-edff8ad0e71a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-fdb355-pv6x5\" (UID: \"29414e40-8647-43d5-972c-edff8ad0e71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fdb355-pv6x5"
Oct 30 19:44:48.756882 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:48.756862    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8hdr\" (UniqueName: \"kubernetes.io/projected/29414e40-8647-43d5-972c-edff8ad0e71a-kube-api-access-x8hdr\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-fdb355-pv6x5\" (UID: \"29414e40-8647-43d5-972c-edff8ad0e71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fdb355-pv6x5"
Oct 30 19:44:48.756947 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:48.756890    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29414e40-8647-43d5-972c-edff8ad0e71a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-fdb355-pv6x5\" (UID: \"29414e40-8647-43d5-972c-edff8ad0e71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fdb355-pv6x5"
Oct 30 19:44:48.858626 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:48.858523    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8hdr\" (UniqueName: \"kubernetes.io/projected/29414e40-8647-43d5-972c-edff8ad0e71a-kube-api-access-x8hdr\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-fdb355-pv6x5\" (UID: \"29414e40-8647-43d5-972c-edff8ad0e71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fdb355-pv6x5"
Oct 30 19:44:48.858626 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:48.858580    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29414e40-8647-43d5-972c-edff8ad0e71a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-fdb355-pv6x5\" (UID: \"29414e40-8647-43d5-972c-edff8ad0e71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fdb355-pv6x5"
Oct 30 19:44:48.858912 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:48.858789    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29414e40-8647-43d5-972c-edff8ad0e71a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-fdb355-pv6x5\" (UID: \"29414e40-8647-43d5-972c-edff8ad0e71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fdb355-pv6x5"
Oct 30 19:44:48.859561 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:48.859535    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29414e40-8647-43d5-972c-edff8ad0e71a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-fdb355-pv6x5\" (UID: \"29414e40-8647-43d5-972c-edff8ad0e71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fdb355-pv6x5"
Oct 30 19:44:48.861380 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:48.861354    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29414e40-8647-43d5-972c-edff8ad0e71a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-fdb355-pv6x5\" (UID: \"29414e40-8647-43d5-972c-edff8ad0e71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fdb355-pv6x5"
Oct 30 19:44:48.871841 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:48.871814    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaae805a-m4l2s"]
Oct 30 19:44:48.872414 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:44:48.872388    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14a645f0_5289_46a9_ab64_2743663bb5d0.slice/crio-8b8dfcb62303578f5c01d181d61dc2b0a86c0baff8a1dae4b0f5f20858d470e4 WatchSource:0}: Error finding container 8b8dfcb62303578f5c01d181d61dc2b0a86c0baff8a1dae4b0f5f20858d470e4: Status 404 returned error can't find the container with id 8b8dfcb62303578f5c01d181d61dc2b0a86c0baff8a1dae4b0f5f20858d470e4
Oct 30 19:44:48.882333 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:48.882305    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8hdr\" (UniqueName: \"kubernetes.io/projected/29414e40-8647-43d5-972c-edff8ad0e71a-kube-api-access-x8hdr\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-fdb355-pv6x5\" (UID: \"29414e40-8647-43d5-972c-edff8ad0e71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fdb355-pv6x5"
Oct 30 19:44:49.047603 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:49.047573    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fdb355-pv6x5"
Oct 30 19:44:49.336062 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:49.336033    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fdb355-pv6x5"]
Oct 30 19:44:49.337303 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:44:49.337273    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29414e40_8647_43d5_972c_edff8ad0e71a.slice/crio-d23f8934161bcbb69471e2bdb3dbbe8c5a7935fa86c24b9b5bf65a46645d4813 WatchSource:0}: Error finding container d23f8934161bcbb69471e2bdb3dbbe8c5a7935fa86c24b9b5bf65a46645d4813: Status 404 returned error can't find the container with id d23f8934161bcbb69471e2bdb3dbbe8c5a7935fa86c24b9b5bf65a46645d4813
Oct 30 19:44:49.807867 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:49.807833    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaae805a-m4l2s" event={"ID":"14a645f0-5289-46a9-ab64-2743663bb5d0","Type":"ContainerStarted","Data":"f1c43b6ecb45bd3c2addb9ef0f16d606d5df54076c18fb58fb53dd06525d4b9a"}
Oct 30 19:44:49.808038 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:49.807895    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaae805a-m4l2s" event={"ID":"14a645f0-5289-46a9-ab64-2743663bb5d0","Type":"ContainerStarted","Data":"8b8dfcb62303578f5c01d181d61dc2b0a86c0baff8a1dae4b0f5f20858d470e4"}
Oct 30 19:44:49.809956 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:49.809929    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fdb355-pv6x5" event={"ID":"29414e40-8647-43d5-972c-edff8ad0e71a","Type":"ContainerStarted","Data":"5fa07d580300dc27006927ed3066c57b1751ece35b0bef03619e29697dc9ac02"}
Oct 30 19:44:49.810078 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:49.809981    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fdb355-pv6x5" event={"ID":"29414e40-8647-43d5-972c-edff8ad0e71a","Type":"ContainerStarted","Data":"d23f8934161bcbb69471e2bdb3dbbe8c5a7935fa86c24b9b5bf65a46645d4813"}
Oct 30 19:44:49.855951 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:49.855891    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaae805a-m4l2s" podStartSLOduration=2.855878279 podStartE2EDuration="2.855878279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:49.852610309 +0000 UTC m=+1709.363670279" watchObservedRunningTime="2026-10-30 19:44:49.855878279 +0000 UTC m=+1709.366938243"
Oct 30 19:44:49.899780 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:49.899693    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fdb355-pv6x5" podStartSLOduration=1.8996730309999998 podStartE2EDuration="1.899673031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:49.892229114 +0000 UTC m=+1709.403289078" watchObservedRunningTime="2026-10-30 19:44:49.899673031 +0000 UTC m=+1709.410732997"
Oct 30 19:44:50.820810 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:50.820773    5043 generic.go:356] "Generic (PLEG): container finished" podID="14a645f0-5289-46a9-ab64-2743663bb5d0" containerID="f1c43b6ecb45bd3c2addb9ef0f16d606d5df54076c18fb58fb53dd06525d4b9a" exitCode=0
Oct 30 19:44:50.821307 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:50.820862    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaae805a-m4l2s" event={"ID":"14a645f0-5289-46a9-ab64-2743663bb5d0","Type":"ContainerDied","Data":"f1c43b6ecb45bd3c2addb9ef0f16d606d5df54076c18fb58fb53dd06525d4b9a"}
Oct 30 19:44:50.822935 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:50.822906    5043 generic.go:356] "Generic (PLEG): container finished" podID="29414e40-8647-43d5-972c-edff8ad0e71a" containerID="5fa07d580300dc27006927ed3066c57b1751ece35b0bef03619e29697dc9ac02" exitCode=0
Oct 30 19:44:50.823079 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:50.822943    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fdb355-pv6x5" event={"ID":"29414e40-8647-43d5-972c-edff8ad0e71a","Type":"ContainerDied","Data":"5fa07d580300dc27006927ed3066c57b1751ece35b0bef03619e29697dc9ac02"}
Oct 30 19:44:52.013659 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:52.012876    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fdb355-pv6x5"
Oct 30 19:44:52.096068 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:52.095996    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaae805a-m4l2s"
Oct 30 19:44:52.104843 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:52.104813    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29414e40-8647-43d5-972c-edff8ad0e71a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29414e40-8647-43d5-972c-edff8ad0e71a-maas-api-openshift-ca\") pod \"29414e40-8647-43d5-972c-edff8ad0e71a\" (UID: \"29414e40-8647-43d5-972c-edff8ad0e71a\") "
Oct 30 19:44:52.104995 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:52.104864    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29414e40-8647-43d5-972c-edff8ad0e71a-kube-api-access-x8hdr\" (UniqueName: \"kubernetes.io/projected/29414e40-8647-43d5-972c-edff8ad0e71a-kube-api-access-x8hdr\") pod \"29414e40-8647-43d5-972c-edff8ad0e71a\" (UID: \"29414e40-8647-43d5-972c-edff8ad0e71a\") "
Oct 30 19:44:52.105073 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:52.105019    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29414e40-8647-43d5-972c-edff8ad0e71a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29414e40-8647-43d5-972c-edff8ad0e71a-maas-api-xks-ca\") pod \"29414e40-8647-43d5-972c-edff8ad0e71a\" (UID: \"29414e40-8647-43d5-972c-edff8ad0e71a\") "
Oct 30 19:44:52.105138 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:52.105076    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29414e40-8647-43d5-972c-edff8ad0e71a-maas-api-openshift-ca" pod "29414e40-8647-43d5-972c-edff8ad0e71a" (UID: "29414e40-8647-43d5-972c-edff8ad0e71a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:52.105610 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:52.105587    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29414e40-8647-43d5-972c-edff8ad0e71a-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:52.107625 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:52.107600    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29414e40-8647-43d5-972c-edff8ad0e71a-kube-api-access-x8hdr" pod "29414e40-8647-43d5-972c-edff8ad0e71a" (UID: "29414e40-8647-43d5-972c-edff8ad0e71a"). InnerVolumeSpecName "kube-api-access-x8hdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:52.107769 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:52.107695    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29414e40-8647-43d5-972c-edff8ad0e71a-maas-api-xks-ca" pod "29414e40-8647-43d5-972c-edff8ad0e71a" (UID: "29414e40-8647-43d5-972c-edff8ad0e71a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:44:52.206594 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:52.206558    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14a645f0-5289-46a9-ab64-2743663bb5d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14a645f0-5289-46a9-ab64-2743663bb5d0-maas-api-xks-ca\") pod \"14a645f0-5289-46a9-ab64-2743663bb5d0\" (UID: \"14a645f0-5289-46a9-ab64-2743663bb5d0\") "
Oct 30 19:44:52.206818 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:52.206643    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14a645f0-5289-46a9-ab64-2743663bb5d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14a645f0-5289-46a9-ab64-2743663bb5d0-maas-api-openshift-ca\") pod \"14a645f0-5289-46a9-ab64-2743663bb5d0\" (UID: \"14a645f0-5289-46a9-ab64-2743663bb5d0\") "
Oct 30 19:44:52.206873 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:52.206814    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14a645f0-5289-46a9-ab64-2743663bb5d0-kube-api-access-s8lrz\" (UniqueName: \"kubernetes.io/projected/14a645f0-5289-46a9-ab64-2743663bb5d0-kube-api-access-s8lrz\") pod \"14a645f0-5289-46a9-ab64-2743663bb5d0\" (UID: \"14a645f0-5289-46a9-ab64-2743663bb5d0\") "
Oct 30 19:44:52.207033 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:52.207007    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14a645f0-5289-46a9-ab64-2743663bb5d0-maas-api-openshift-ca" pod "14a645f0-5289-46a9-ab64-2743663bb5d0" (UID: "14a645f0-5289-46a9-ab64-2743663bb5d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:52.207279 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:52.207259    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29414e40-8647-43d5-972c-edff8ad0e71a-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:52.207279 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:52.207278    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14a645f0-5289-46a9-ab64-2743663bb5d0-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:52.207475 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:52.207289    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8hdr\" (UniqueName: \"kubernetes.io/projected/29414e40-8647-43d5-972c-edff8ad0e71a-kube-api-access-x8hdr\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:52.208938 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:52.208915    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14a645f0-5289-46a9-ab64-2743663bb5d0-maas-api-xks-ca" pod "14a645f0-5289-46a9-ab64-2743663bb5d0" (UID: "14a645f0-5289-46a9-ab64-2743663bb5d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:44:52.209035 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:52.209017    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14a645f0-5289-46a9-ab64-2743663bb5d0-kube-api-access-s8lrz" pod "14a645f0-5289-46a9-ab64-2743663bb5d0" (UID: "14a645f0-5289-46a9-ab64-2743663bb5d0"). InnerVolumeSpecName "kube-api-access-s8lrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:52.308471 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:52.308439    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14a645f0-5289-46a9-ab64-2743663bb5d0-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:52.308471 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:52.308468    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8lrz\" (UniqueName: \"kubernetes.io/projected/14a645f0-5289-46a9-ab64-2743663bb5d0-kube-api-access-s8lrz\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:52.840267 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:52.840239    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaae805a-m4l2s"
Oct 30 19:44:52.840479 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:52.840284    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaae805a-m4l2s" event={"ID":"14a645f0-5289-46a9-ab64-2743663bb5d0","Type":"ContainerDied","Data":"8b8dfcb62303578f5c01d181d61dc2b0a86c0baff8a1dae4b0f5f20858d470e4"}
Oct 30 19:44:52.840479 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:52.840340    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b8dfcb62303578f5c01d181d61dc2b0a86c0baff8a1dae4b0f5f20858d470e4"
Oct 30 19:44:52.841843 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:52.841824    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fdb355-pv6x5"
Oct 30 19:44:52.841948 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:52.841866    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fdb355-pv6x5" event={"ID":"29414e40-8647-43d5-972c-edff8ad0e71a","Type":"ContainerDied","Data":"d23f8934161bcbb69471e2bdb3dbbe8c5a7935fa86c24b9b5bf65a46645d4813"}
Oct 30 19:44:52.841948 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:52.841906    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d23f8934161bcbb69471e2bdb3dbbe8c5a7935fa86c24b9b5bf65a46645d4813"
Oct 30 19:44:53.371206 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.371181    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r"
Oct 30 19:44:53.422255 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.422223    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76b4e8e9-54ac-4461-b99c-a6f0361505b6-istio-token\" (UniqueName: \"kubernetes.io/projected/76b4e8e9-54ac-4461-b99c-a6f0361505b6-istio-token\") pod \"76b4e8e9-54ac-4461-b99c-a6f0361505b6\" (UID: \"76b4e8e9-54ac-4461-b99c-a6f0361505b6\") "
Oct 30 19:44:53.422446 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.422342    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76b4e8e9-54ac-4461-b99c-a6f0361505b6-kube-api-access-64tgh\" (UniqueName: \"kubernetes.io/projected/76b4e8e9-54ac-4461-b99c-a6f0361505b6-kube-api-access-64tgh\") pod \"76b4e8e9-54ac-4461-b99c-a6f0361505b6\" (UID: \"76b4e8e9-54ac-4461-b99c-a6f0361505b6\") "
Oct 30 19:44:53.422446 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.422377    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76b4e8e9-54ac-4461-b99c-a6f0361505b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76b4e8e9-54ac-4461-b99c-a6f0361505b6-workload-certs\") pod \"76b4e8e9-54ac-4461-b99c-a6f0361505b6\" (UID: \"76b4e8e9-54ac-4461-b99c-a6f0361505b6\") "
Oct 30 19:44:53.422446 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.422419    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76b4e8e9-54ac-4461-b99c-a6f0361505b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76b4e8e9-54ac-4461-b99c-a6f0361505b6-istio-data\") pod \"76b4e8e9-54ac-4461-b99c-a6f0361505b6\" (UID: \"76b4e8e9-54ac-4461-b99c-a6f0361505b6\") "
Oct 30 19:44:53.422677 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.422459    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76b4e8e9-54ac-4461-b99c-a6f0361505b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76b4e8e9-54ac-4461-b99c-a6f0361505b6-istio-envoy\") pod \"76b4e8e9-54ac-4461-b99c-a6f0361505b6\" (UID: \"76b4e8e9-54ac-4461-b99c-a6f0361505b6\") "
Oct 30 19:44:53.422677 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.422481    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76b4e8e9-54ac-4461-b99c-a6f0361505b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76b4e8e9-54ac-4461-b99c-a6f0361505b6-istiod-ca-cert\") pod \"76b4e8e9-54ac-4461-b99c-a6f0361505b6\" (UID: \"76b4e8e9-54ac-4461-b99c-a6f0361505b6\") "
Oct 30 19:44:53.422677 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.422524    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76b4e8e9-54ac-4461-b99c-a6f0361505b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76b4e8e9-54ac-4461-b99c-a6f0361505b6-istio-podinfo\") pod \"76b4e8e9-54ac-4461-b99c-a6f0361505b6\" (UID: \"76b4e8e9-54ac-4461-b99c-a6f0361505b6\") "
Oct 30 19:44:53.422677 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.422558    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76b4e8e9-54ac-4461-b99c-a6f0361505b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b4e8e9-54ac-4461-b99c-a6f0361505b6-workload-socket\") pod \"76b4e8e9-54ac-4461-b99c-a6f0361505b6\" (UID: \"76b4e8e9-54ac-4461-b99c-a6f0361505b6\") "
Oct 30 19:44:53.422677 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.422614    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76b4e8e9-54ac-4461-b99c-a6f0361505b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b4e8e9-54ac-4461-b99c-a6f0361505b6-credential-socket\") pod \"76b4e8e9-54ac-4461-b99c-a6f0361505b6\" (UID: \"76b4e8e9-54ac-4461-b99c-a6f0361505b6\") "
Oct 30 19:44:53.422982 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.422750    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b4e8e9-54ac-4461-b99c-a6f0361505b6-workload-certs" pod "76b4e8e9-54ac-4461-b99c-a6f0361505b6" (UID: "76b4e8e9-54ac-4461-b99c-a6f0361505b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:53.422982 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.422940    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76b4e8e9-54ac-4461-b99c-a6f0361505b6-istiod-ca-cert" pod "76b4e8e9-54ac-4461-b99c-a6f0361505b6" (UID: "76b4e8e9-54ac-4461-b99c-a6f0361505b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:53.423196 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.423158    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b4e8e9-54ac-4461-b99c-a6f0361505b6-credential-socket" pod "76b4e8e9-54ac-4461-b99c-a6f0361505b6" (UID: "76b4e8e9-54ac-4461-b99c-a6f0361505b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:53.423402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.423247    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b4e8e9-54ac-4461-b99c-a6f0361505b6-workload-socket" pod "76b4e8e9-54ac-4461-b99c-a6f0361505b6" (UID: "76b4e8e9-54ac-4461-b99c-a6f0361505b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:53.423402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.423341    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b4e8e9-54ac-4461-b99c-a6f0361505b6-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:53.423402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.423355    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b4e8e9-54ac-4461-b99c-a6f0361505b6-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:53.423402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.423364    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76b4e8e9-54ac-4461-b99c-a6f0361505b6-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:53.423402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.423378    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76b4e8e9-54ac-4461-b99c-a6f0361505b6-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:53.423688 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.423438    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b4e8e9-54ac-4461-b99c-a6f0361505b6-istio-data" pod "76b4e8e9-54ac-4461-b99c-a6f0361505b6" (UID: "76b4e8e9-54ac-4461-b99c-a6f0361505b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:53.425300 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.425273    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76b4e8e9-54ac-4461-b99c-a6f0361505b6-istio-token" pod "76b4e8e9-54ac-4461-b99c-a6f0361505b6" (UID: "76b4e8e9-54ac-4461-b99c-a6f0361505b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:53.425508 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.425477    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b4e8e9-54ac-4461-b99c-a6f0361505b6-istio-envoy" pod "76b4e8e9-54ac-4461-b99c-a6f0361505b6" (UID: "76b4e8e9-54ac-4461-b99c-a6f0361505b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:53.425633 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.425593    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76b4e8e9-54ac-4461-b99c-a6f0361505b6-kube-api-access-64tgh" pod "76b4e8e9-54ac-4461-b99c-a6f0361505b6" (UID: "76b4e8e9-54ac-4461-b99c-a6f0361505b6"). InnerVolumeSpecName "kube-api-access-64tgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:53.425947 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.425917    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76b4e8e9-54ac-4461-b99c-a6f0361505b6-istio-podinfo" pod "76b4e8e9-54ac-4461-b99c-a6f0361505b6" (UID: "76b4e8e9-54ac-4461-b99c-a6f0361505b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:44:53.524571 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.524477    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76b4e8e9-54ac-4461-b99c-a6f0361505b6-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:53.524571 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.524509    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76b4e8e9-54ac-4461-b99c-a6f0361505b6-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:53.524571 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.524519    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76b4e8e9-54ac-4461-b99c-a6f0361505b6-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:53.524571 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.524529    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64tgh\" (UniqueName: \"kubernetes.io/projected/76b4e8e9-54ac-4461-b99c-a6f0361505b6-kube-api-access-64tgh\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:53.524571 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.524540    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76b4e8e9-54ac-4461-b99c-a6f0361505b6-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:53.811949 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.811924    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ac37c8-57cb47d8c-4cprd"
Oct 30 19:44:53.876765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.875479    5043 generic.go:356] "Generic (PLEG): container finished" podID="76b4e8e9-54ac-4461-b99c-a6f0361505b6" containerID="8308f4a0b3eb68229f823b999921178a37ea37806d32fd6f180b18d02137155e" exitCode=0
Oct 30 19:44:53.876765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.875582    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r" event={"ID":"76b4e8e9-54ac-4461-b99c-a6f0361505b6","Type":"ContainerDied","Data":"8308f4a0b3eb68229f823b999921178a37ea37806d32fd6f180b18d02137155e"}
Oct 30 19:44:53.876765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.875634    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r" event={"ID":"76b4e8e9-54ac-4461-b99c-a6f0361505b6","Type":"ContainerDied","Data":"2b6f80d3e81669eaacc253f7e90c31c1d3d42914e8f39813703931a754646cdc"}
Oct 30 19:44:53.876765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.875667    5043 scope.go:122] "RemoveContainer" containerID="8308f4a0b3eb68229f823b999921178a37ea37806d32fd6f180b18d02137155e"
Oct 30 19:44:53.876765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.875870    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r"
Oct 30 19:44:53.898796 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.898228    5043 generic.go:356] "Generic (PLEG): container finished" podID="42cfcd85-0bda-4e33-ae8c-f84eb90afd6b" containerID="a85ae4b85ef525013d42625b7cbae6c62a0d89dfa40311c97d4094706e64536e" exitCode=0
Oct 30 19:44:53.898796 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.898444    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ac37c8-57cb47d8c-4cprd" event={"ID":"42cfcd85-0bda-4e33-ae8c-f84eb90afd6b","Type":"ContainerDied","Data":"a85ae4b85ef525013d42625b7cbae6c62a0d89dfa40311c97d4094706e64536e"}
Oct 30 19:44:53.898796 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.898489    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ac37c8-57cb47d8c-4cprd" event={"ID":"42cfcd85-0bda-4e33-ae8c-f84eb90afd6b","Type":"ContainerDied","Data":"2b4c521f80eb333941f899463e85f28336df8cbff7267130d349dd545d46c928"}
Oct 30 19:44:53.898796 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.898561    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ac37c8-57cb47d8c-4cprd"
Oct 30 19:44:53.946412 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.944874    5043 scope.go:122] "RemoveContainer" containerID="17ea88734869b20fec43ff49cb62bc641cda682522f45e6b64cc406eaeb3daf5"
Oct 30 19:44:53.951833 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.950954    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42cfcd85-0bda-4e33-ae8c-f84eb90afd6b-kube-api-access-nfx66\" (UniqueName: \"kubernetes.io/projected/42cfcd85-0bda-4e33-ae8c-f84eb90afd6b-kube-api-access-nfx66\") pod \"42cfcd85-0bda-4e33-ae8c-f84eb90afd6b\" (UID: \"42cfcd85-0bda-4e33-ae8c-f84eb90afd6b\") "
Oct 30 19:44:53.951833 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.951316    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42cfcd85-0bda-4e33-ae8c-f84eb90afd6b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42cfcd85-0bda-4e33-ae8c-f84eb90afd6b-plugins-config-volume\") pod \"42cfcd85-0bda-4e33-ae8c-f84eb90afd6b\" (UID: \"42cfcd85-0bda-4e33-ae8c-f84eb90afd6b\") "
Oct 30 19:44:53.954701 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.954669    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42cfcd85-0bda-4e33-ae8c-f84eb90afd6b-plugins-config-volume" pod "42cfcd85-0bda-4e33-ae8c-f84eb90afd6b" (UID: "42cfcd85-0bda-4e33-ae8c-f84eb90afd6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:53.958144 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.958100    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42cfcd85-0bda-4e33-ae8c-f84eb90afd6b-kube-api-access-nfx66" pod "42cfcd85-0bda-4e33-ae8c-f84eb90afd6b" (UID: "42cfcd85-0bda-4e33-ae8c-f84eb90afd6b"). InnerVolumeSpecName "kube-api-access-nfx66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:53.959392 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.959286    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r"]
Oct 30 19:44:53.964279 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.964257    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r"]
Oct 30 19:44:53.968223 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.968201    5043 scope.go:122] "RemoveContainer" containerID="8308f4a0b3eb68229f823b999921178a37ea37806d32fd6f180b18d02137155e"
Oct 30 19:44:53.968598 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:44:53.968568    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8308f4a0b3eb68229f823b999921178a37ea37806d32fd6f180b18d02137155e\": container with ID starting with 8308f4a0b3eb68229f823b999921178a37ea37806d32fd6f180b18d02137155e not found: ID does not exist" containerID="8308f4a0b3eb68229f823b999921178a37ea37806d32fd6f180b18d02137155e"
Oct 30 19:44:53.968668 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.968607    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8308f4a0b3eb68229f823b999921178a37ea37806d32fd6f180b18d02137155e"} err="failed to get container status \"8308f4a0b3eb68229f823b999921178a37ea37806d32fd6f180b18d02137155e\": rpc error: code = NotFound desc = could not find container \"8308f4a0b3eb68229f823b999921178a37ea37806d32fd6f180b18d02137155e\": container with ID starting with 8308f4a0b3eb68229f823b999921178a37ea37806d32fd6f180b18d02137155e not found: ID does not exist"
Oct 30 19:44:53.968668 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.968625    5043 scope.go:122] "RemoveContainer" containerID="17ea88734869b20fec43ff49cb62bc641cda682522f45e6b64cc406eaeb3daf5"
Oct 30 19:44:53.969082 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:44:53.969058    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17ea88734869b20fec43ff49cb62bc641cda682522f45e6b64cc406eaeb3daf5\": container with ID starting with 17ea88734869b20fec43ff49cb62bc641cda682522f45e6b64cc406eaeb3daf5 not found: ID does not exist" containerID="17ea88734869b20fec43ff49cb62bc641cda682522f45e6b64cc406eaeb3daf5"
Oct 30 19:44:53.969180 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.969094    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17ea88734869b20fec43ff49cb62bc641cda682522f45e6b64cc406eaeb3daf5"} err="failed to get container status \"17ea88734869b20fec43ff49cb62bc641cda682522f45e6b64cc406eaeb3daf5\": rpc error: code = NotFound desc = could not find container \"17ea88734869b20fec43ff49cb62bc641cda682522f45e6b64cc406eaeb3daf5\": container with ID starting with 17ea88734869b20fec43ff49cb62bc641cda682522f45e6b64cc406eaeb3daf5 not found: ID does not exist"
Oct 30 19:44:53.969180 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.969118    5043 scope.go:122] "RemoveContainer" containerID="a85ae4b85ef525013d42625b7cbae6c62a0d89dfa40311c97d4094706e64536e"
Oct 30 19:44:53.986979 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.986959    5043 scope.go:122] "RemoveContainer" containerID="a85ae4b85ef525013d42625b7cbae6c62a0d89dfa40311c97d4094706e64536e"
Oct 30 19:44:53.987355 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:44:53.987329    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a85ae4b85ef525013d42625b7cbae6c62a0d89dfa40311c97d4094706e64536e\": container with ID starting with a85ae4b85ef525013d42625b7cbae6c62a0d89dfa40311c97d4094706e64536e not found: ID does not exist" containerID="a85ae4b85ef525013d42625b7cbae6c62a0d89dfa40311c97d4094706e64536e"
Oct 30 19:44:53.987446 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:53.987362    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a85ae4b85ef525013d42625b7cbae6c62a0d89dfa40311c97d4094706e64536e"} err="failed to get container status \"a85ae4b85ef525013d42625b7cbae6c62a0d89dfa40311c97d4094706e64536e\": rpc error: code = NotFound desc = could not find container \"a85ae4b85ef525013d42625b7cbae6c62a0d89dfa40311c97d4094706e64536e\": container with ID starting with a85ae4b85ef525013d42625b7cbae6c62a0d89dfa40311c97d4094706e64536e not found: ID does not exist"
Oct 30 19:44:54.052816 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:54.052774    5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42cfcd85-0bda-4e33-ae8c-f84eb90afd6b-plugins-config-volume\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:54.052816 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:54.052814    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfx66\" (UniqueName: \"kubernetes.io/projected/42cfcd85-0bda-4e33-ae8c-f84eb90afd6b-kube-api-access-nfx66\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:54.225641 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:54.225608    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ac37c8-57cb47d8c-4cprd"]
Oct 30 19:44:54.228943 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:54.228922    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ac37c8-57cb47d8c-4cprd"]
Oct 30 19:44:54.302296 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:54.302252    5043 patch_prober.go:29] interesting pod/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:44:54.302492 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:54.302318    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-qj56r" podUID="76b4e8e9-54ac-4461-b99c-a6f0361505b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:54.456226 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:54.456195    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-419d0a-7hw77"]
Oct 30 19:44:54.456918 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:54.456899    5043 cpu_manager.go:414] "containerMap: removing container" podUID="29414e40-8647-43d5-972c-edff8ad0e71a" containerName="revoke-keys"
Oct 30 19:44:54.456987 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:54.456921    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29414e40-8647-43d5-972c-edff8ad0e71a" containerName="revoke-keys"
Oct 30 19:44:54.456987 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:54.456966    5043 cpu_manager.go:414] "containerMap: removing container" podUID="76b4e8e9-54ac-4461-b99c-a6f0361505b6" containerName="istio-proxy"
Oct 30 19:44:54.456987 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:54.456973    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76b4e8e9-54ac-4461-b99c-a6f0361505b6" containerName="istio-proxy"
Oct 30 19:44:54.457086 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:54.457010    5043 cpu_manager.go:414] "containerMap: removing container" podUID="14a645f0-5289-46a9-ab64-2743663bb5d0" containerName="revoke-keys"
Oct 30 19:44:54.457086 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:54.457022    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14a645f0-5289-46a9-ab64-2743663bb5d0" containerName="revoke-keys"
Oct 30 19:44:54.457086 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:54.457033    5043 cpu_manager.go:414] "containerMap: removing container" podUID="76b4e8e9-54ac-4461-b99c-a6f0361505b6" containerName="istio-proxy"
Oct 30 19:44:54.457086 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:54.457041    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76b4e8e9-54ac-4461-b99c-a6f0361505b6" containerName="istio-proxy"
Oct 30 19:44:54.457086 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:54.457051    5043 cpu_manager.go:414] "containerMap: removing container" podUID="42cfcd85-0bda-4e33-ae8c-f84eb90afd6b" containerName="payload-pre-processing"
Oct 30 19:44:54.457086 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:54.457058    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42cfcd85-0bda-4e33-ae8c-f84eb90afd6b" containerName="payload-pre-processing"
Oct 30 19:44:54.457270 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:54.457202    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="29414e40-8647-43d5-972c-edff8ad0e71a" containerName="revoke-keys"
Oct 30 19:44:54.457270 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:54.457218    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="76b4e8e9-54ac-4461-b99c-a6f0361505b6" containerName="istio-proxy"
Oct 30 19:44:54.457270 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:54.457231    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="14a645f0-5289-46a9-ab64-2743663bb5d0" containerName="revoke-keys"
Oct 30 19:44:54.457270 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:54.457242    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="42cfcd85-0bda-4e33-ae8c-f84eb90afd6b" containerName="payload-pre-processing"
Oct 30 19:44:54.462158 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:54.462143    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-419d0a-7hw77"
Oct 30 19:44:54.474632 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:54.474609    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:44:54.474891 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:54.474873    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:44:54.479813 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:54.479792    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-419d0a-7hw77"]
Oct 30 19:44:54.549033 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:54.548996    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:44:54.549181 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:54.549052    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:44:54.558369 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:54.558340    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-722jc\" (UniqueName: \"kubernetes.io/projected/0bf48130-92b3-4703-8a2a-4bc57504a3c9-kube-api-access-722jc\") pod \"maas-api-revoke-keys-e2e-worker-w1-419d0a-7hw77\" (UID: \"0bf48130-92b3-4703-8a2a-4bc57504a3c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-419d0a-7hw77"
Oct 30 19:44:54.558505 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:54.558488    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bf48130-92b3-4703-8a2a-4bc57504a3c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-419d0a-7hw77\" (UID: \"0bf48130-92b3-4703-8a2a-4bc57504a3c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-419d0a-7hw77"
Oct 30 19:44:54.558582 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:54.558564    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bf48130-92b3-4703-8a2a-4bc57504a3c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-419d0a-7hw77\" (UID: \"0bf48130-92b3-4703-8a2a-4bc57504a3c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-419d0a-7hw77"
Oct 30 19:44:54.646374 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:54.646339    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42cfcd85-0bda-4e33-ae8c-f84eb90afd6b" path="/var/lib/kubelet/pods/42cfcd85-0bda-4e33-ae8c-f84eb90afd6b/volumes"
Oct 30 19:44:54.646968 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:54.646943    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76b4e8e9-54ac-4461-b99c-a6f0361505b6" path="/var/lib/kubelet/pods/76b4e8e9-54ac-4461-b99c-a6f0361505b6/volumes"
Oct 30 19:44:54.660232 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:54.660196    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-722jc\" (UniqueName: \"kubernetes.io/projected/0bf48130-92b3-4703-8a2a-4bc57504a3c9-kube-api-access-722jc\") pod \"maas-api-revoke-keys-e2e-worker-w1-419d0a-7hw77\" (UID: \"0bf48130-92b3-4703-8a2a-4bc57504a3c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-419d0a-7hw77"
Oct 30 19:44:54.660363 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:54.660321    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bf48130-92b3-4703-8a2a-4bc57504a3c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-419d0a-7hw77\" (UID: \"0bf48130-92b3-4703-8a2a-4bc57504a3c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-419d0a-7hw77"
Oct 30 19:44:54.660423 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:54.660411    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bf48130-92b3-4703-8a2a-4bc57504a3c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-419d0a-7hw77\" (UID: \"0bf48130-92b3-4703-8a2a-4bc57504a3c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-419d0a-7hw77"
Oct 30 19:44:54.660996 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:54.660977    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bf48130-92b3-4703-8a2a-4bc57504a3c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-419d0a-7hw77\" (UID: \"0bf48130-92b3-4703-8a2a-4bc57504a3c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-419d0a-7hw77"
Oct 30 19:44:54.663583 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:54.663563    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bf48130-92b3-4703-8a2a-4bc57504a3c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-419d0a-7hw77\" (UID: \"0bf48130-92b3-4703-8a2a-4bc57504a3c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-419d0a-7hw77"
Oct 30 19:44:54.669273 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:54.669245    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-722jc\" (UniqueName: \"kubernetes.io/projected/0bf48130-92b3-4703-8a2a-4bc57504a3c9-kube-api-access-722jc\") pod \"maas-api-revoke-keys-e2e-worker-w1-419d0a-7hw77\" (UID: \"0bf48130-92b3-4703-8a2a-4bc57504a3c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-419d0a-7hw77"
Oct 30 19:44:54.776895 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:54.776810    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-419d0a-7hw77"
Oct 30 19:44:54.977285 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:44:54.977238    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:44:55.059013 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:55.058989    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-419d0a-7hw77"]
Oct 30 19:44:55.060958 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:44:55.060929    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bf48130_92b3_4703_8a2a_4bc57504a3c9.slice/crio-b2c96362f10ab0ed9ae82bd67dfe11e4309658dfb491d701da005581dec97612 WatchSource:0}: Error finding container b2c96362f10ab0ed9ae82bd67dfe11e4309658dfb491d701da005581dec97612: Status 404 returned error can't find the container with id b2c96362f10ab0ed9ae82bd67dfe11e4309658dfb491d701da005581dec97612
Oct 30 19:44:55.922418 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:55.922379    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-419d0a-7hw77" event={"ID":"0bf48130-92b3-4703-8a2a-4bc57504a3c9","Type":"ContainerStarted","Data":"d7860678b4b1b3ad0650ebd96fd6145e8586533e2ad0fa417b4476b00834fc07"}
Oct 30 19:44:55.922866 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:55.922471    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-419d0a-7hw77" event={"ID":"0bf48130-92b3-4703-8a2a-4bc57504a3c9","Type":"ContainerStarted","Data":"b2c96362f10ab0ed9ae82bd67dfe11e4309658dfb491d701da005581dec97612"}
Oct 30 19:44:55.944344 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:55.944286    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-419d0a-7hw77" podStartSLOduration=1.944270663 podStartE2EDuration="1.944270663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:55.940817512 +0000 UTC m=+1715.451877518" watchObservedRunningTime="2026-10-30 19:44:55.944270663 +0000 UTC m=+1715.455330610"
Oct 30 19:44:56.932532 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:56.932493    5043 generic.go:356] "Generic (PLEG): container finished" podID="0bf48130-92b3-4703-8a2a-4bc57504a3c9" containerID="d7860678b4b1b3ad0650ebd96fd6145e8586533e2ad0fa417b4476b00834fc07" exitCode=0
Oct 30 19:44:56.932984 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:56.932574    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-419d0a-7hw77" event={"ID":"0bf48130-92b3-4703-8a2a-4bc57504a3c9","Type":"ContainerDied","Data":"d7860678b4b1b3ad0650ebd96fd6145e8586533e2ad0fa417b4476b00834fc07"}
Oct 30 19:44:57.415786 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:57.415720    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-01b24e-9gkpt"]
Oct 30 19:44:57.422730 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:57.422705    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-01b24e-9gkpt"]
Oct 30 19:44:58.099767 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:58.099706    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-419d0a-7hw77"
Oct 30 19:44:58.203457 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:58.203369    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bf48130-92b3-4703-8a2a-4bc57504a3c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bf48130-92b3-4703-8a2a-4bc57504a3c9-maas-api-xks-ca\") pod \"0bf48130-92b3-4703-8a2a-4bc57504a3c9\" (UID: \"0bf48130-92b3-4703-8a2a-4bc57504a3c9\") "
Oct 30 19:44:58.203457 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:58.203433    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bf48130-92b3-4703-8a2a-4bc57504a3c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bf48130-92b3-4703-8a2a-4bc57504a3c9-maas-api-openshift-ca\") pod \"0bf48130-92b3-4703-8a2a-4bc57504a3c9\" (UID: \"0bf48130-92b3-4703-8a2a-4bc57504a3c9\") "
Oct 30 19:44:58.203687 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:58.203568    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bf48130-92b3-4703-8a2a-4bc57504a3c9-kube-api-access-722jc\" (UniqueName: \"kubernetes.io/projected/0bf48130-92b3-4703-8a2a-4bc57504a3c9-kube-api-access-722jc\") pod \"0bf48130-92b3-4703-8a2a-4bc57504a3c9\" (UID: \"0bf48130-92b3-4703-8a2a-4bc57504a3c9\") "
Oct 30 19:44:58.203883 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:58.203851    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bf48130-92b3-4703-8a2a-4bc57504a3c9-maas-api-openshift-ca" pod "0bf48130-92b3-4703-8a2a-4bc57504a3c9" (UID: "0bf48130-92b3-4703-8a2a-4bc57504a3c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:58.204017 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:58.203988    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bf48130-92b3-4703-8a2a-4bc57504a3c9-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:58.205642 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:58.205612    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bf48130-92b3-4703-8a2a-4bc57504a3c9-maas-api-xks-ca" pod "0bf48130-92b3-4703-8a2a-4bc57504a3c9" (UID: "0bf48130-92b3-4703-8a2a-4bc57504a3c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:44:58.205814 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:58.205770    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bf48130-92b3-4703-8a2a-4bc57504a3c9-kube-api-access-722jc" pod "0bf48130-92b3-4703-8a2a-4bc57504a3c9" (UID: "0bf48130-92b3-4703-8a2a-4bc57504a3c9"). InnerVolumeSpecName "kube-api-access-722jc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:58.304897 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:58.304868    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bf48130-92b3-4703-8a2a-4bc57504a3c9-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:58.304897 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:58.304895    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-722jc\" (UniqueName: \"kubernetes.io/projected/0bf48130-92b3-4703-8a2a-4bc57504a3c9-kube-api-access-722jc\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:58.635649 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:58.635609    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-47110f/test-model-47110f-kserve-bd876684f-4x8lh"]
Oct 30 19:44:58.647814 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:58.647719    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00c8d4cc-34b3-4022-8434-f41d5bd176ed" path="/var/lib/kubelet/pods/00c8d4cc-34b3-4022-8434-f41d5bd176ed/volumes"
Oct 30 19:44:58.918986 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:58.918914    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3e35df-xcvqg"]
Oct 30 19:44:58.919457 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:58.919444    5043 cpu_manager.go:414] "containerMap: removing container" podUID="0bf48130-92b3-4703-8a2a-4bc57504a3c9" containerName="revoke-keys"
Oct 30 19:44:58.919501 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:58.919458    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bf48130-92b3-4703-8a2a-4bc57504a3c9" containerName="revoke-keys"
Oct 30 19:44:58.919564 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:58.919556    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="76b4e8e9-54ac-4461-b99c-a6f0361505b6" containerName="istio-proxy"
Oct 30 19:44:58.919604 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:58.919573    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bf48130-92b3-4703-8a2a-4bc57504a3c9" containerName="revoke-keys"
Oct 30 19:44:58.923241 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:58.923225    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3e35df-xcvqg"
Oct 30 19:44:58.948169 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:58.948139    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3e35df-xcvqg"]
Oct 30 19:44:58.950087 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:58.950056    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-419d0a-7hw77" event={"ID":"0bf48130-92b3-4703-8a2a-4bc57504a3c9","Type":"ContainerDied","Data":"b2c96362f10ab0ed9ae82bd67dfe11e4309658dfb491d701da005581dec97612"}
Oct 30 19:44:58.950087 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:58.950074    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-419d0a-7hw77"
Oct 30 19:44:58.950241 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:58.950100    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2c96362f10ab0ed9ae82bd67dfe11e4309658dfb491d701da005581dec97612"
Oct 30 19:44:59.012340 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:59.012309    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dj67\" (UniqueName: \"kubernetes.io/projected/094be84d-0dd7-47d1-87a2-e0935eb38af9-kube-api-access-6dj67\") pod \"maas-api-revoke-keys-e2e-inf-a-3e35df-xcvqg\" (UID: \"094be84d-0dd7-47d1-87a2-e0935eb38af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3e35df-xcvqg"
Oct 30 19:44:59.012506 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:59.012462    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/094be84d-0dd7-47d1-87a2-e0935eb38af9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3e35df-xcvqg\" (UID: \"094be84d-0dd7-47d1-87a2-e0935eb38af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3e35df-xcvqg"
Oct 30 19:44:59.012564 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:59.012549    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/094be84d-0dd7-47d1-87a2-e0935eb38af9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3e35df-xcvqg\" (UID: \"094be84d-0dd7-47d1-87a2-e0935eb38af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3e35df-xcvqg"
Oct 30 19:44:59.113421 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:59.113373    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dj67\" (UniqueName: \"kubernetes.io/projected/094be84d-0dd7-47d1-87a2-e0935eb38af9-kube-api-access-6dj67\") pod \"maas-api-revoke-keys-e2e-inf-a-3e35df-xcvqg\" (UID: \"094be84d-0dd7-47d1-87a2-e0935eb38af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3e35df-xcvqg"
Oct 30 19:44:59.113956 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:59.113520    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/094be84d-0dd7-47d1-87a2-e0935eb38af9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3e35df-xcvqg\" (UID: \"094be84d-0dd7-47d1-87a2-e0935eb38af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3e35df-xcvqg"
Oct 30 19:44:59.113956 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:59.113613    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/094be84d-0dd7-47d1-87a2-e0935eb38af9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3e35df-xcvqg\" (UID: \"094be84d-0dd7-47d1-87a2-e0935eb38af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3e35df-xcvqg"
Oct 30 19:44:59.114249 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:59.114225    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/094be84d-0dd7-47d1-87a2-e0935eb38af9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3e35df-xcvqg\" (UID: \"094be84d-0dd7-47d1-87a2-e0935eb38af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3e35df-xcvqg"
Oct 30 19:44:59.116357 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:59.116329    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/094be84d-0dd7-47d1-87a2-e0935eb38af9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3e35df-xcvqg\" (UID: \"094be84d-0dd7-47d1-87a2-e0935eb38af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3e35df-xcvqg"
Oct 30 19:44:59.124608 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:59.124583    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dj67\" (UniqueName: \"kubernetes.io/projected/094be84d-0dd7-47d1-87a2-e0935eb38af9-kube-api-access-6dj67\") pod \"maas-api-revoke-keys-e2e-inf-a-3e35df-xcvqg\" (UID: \"094be84d-0dd7-47d1-87a2-e0935eb38af9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3e35df-xcvqg"
Oct 30 19:44:59.240974 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:59.240928    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3e35df-xcvqg"
Oct 30 19:44:59.542871 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:59.542839    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3e35df-xcvqg"]
Oct 30 19:44:59.543772 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:44:59.543728    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod094be84d_0dd7_47d1_87a2_e0935eb38af9.slice/crio-b431ed979dd15213a77ba97bb696acc2f3a13426b29854befbb46b487e7b9052 WatchSource:0}: Error finding container b431ed979dd15213a77ba97bb696acc2f3a13426b29854befbb46b487e7b9052: Status 404 returned error can't find the container with id b431ed979dd15213a77ba97bb696acc2f3a13426b29854befbb46b487e7b9052
Oct 30 19:44:59.960155 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:44:59.960118    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3e35df-xcvqg" event={"ID":"094be84d-0dd7-47d1-87a2-e0935eb38af9","Type":"ContainerStarted","Data":"b431ed979dd15213a77ba97bb696acc2f3a13426b29854befbb46b487e7b9052"}
Oct 30 19:45:00.152302 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.152272    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fdb355-29889825-nzls2"]
Oct 30 19:45:00.156472 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.156421    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fdb355-29889825-nzls2"
Oct 30 19:45:00.160017 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.159990    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-rm8xg"]
Oct 30 19:45:00.164774 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.164753    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-rm8xg"
Oct 30 19:45:00.181498 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.180241    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889825-ntqm4"]
Oct 30 19:45:00.207244 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.207209    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889825-ntqm4"
Oct 30 19:45:00.211493 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.211403    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fdb355-29889825-nzls2"]
Oct 30 19:45:00.219833 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.219804    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889825-8j66w"]
Oct 30 19:45:00.229200 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.226173    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-rm8xg"]
Oct 30 19:45:00.229200 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.226280    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889825-8j66w"
Oct 30 19:45:00.230330 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.230307    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp42h\" (UniqueName: \"kubernetes.io/projected/834c26fa-e19d-4db2-92ea-331d49a139aa-kube-api-access-xp42h\") pod \"maas-api-key-cleanup-29889825-rm8xg\" (UID: \"834c26fa-e19d-4db2-92ea-331d49a139aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-rm8xg"
Oct 30 19:45:00.230440 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.230345    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/834c26fa-e19d-4db2-92ea-331d49a139aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889825-rm8xg\" (UID: \"834c26fa-e19d-4db2-92ea-331d49a139aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-rm8xg"
Oct 30 19:45:00.230440 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.230400    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7bgl\" (UniqueName: \"kubernetes.io/projected/b62d96d6-6b2d-41e3-add9-bf09e1ca9407-kube-api-access-k7bgl\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-fdb355-29889825-nzls2\" (UID: \"b62d96d6-6b2d-41e3-add9-bf09e1ca9407\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fdb355-29889825-nzls2"
Oct 30 19:45:00.230613 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.230445    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b62d96d6-6b2d-41e3-add9-bf09e1ca9407-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-fdb355-29889825-nzls2\" (UID: \"b62d96d6-6b2d-41e3-add9-bf09e1ca9407\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fdb355-29889825-nzls2"
Oct 30 19:45:00.232969 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.232916    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889825-ntqm4"]
Oct 30 19:45:00.237348 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.237300    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889825-8j66w"]
Oct 30 19:45:00.241695 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.241576    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889825-bz9lm"]
Oct 30 19:45:00.247446 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.247422    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889825-bz9lm"]
Oct 30 19:45:00.247571 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.247526    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889825-bz9lm"
Oct 30 19:45:00.332124 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.332088    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp42h\" (UniqueName: \"kubernetes.io/projected/834c26fa-e19d-4db2-92ea-331d49a139aa-kube-api-access-xp42h\") pod \"maas-api-key-cleanup-29889825-rm8xg\" (UID: \"834c26fa-e19d-4db2-92ea-331d49a139aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-rm8xg"
Oct 30 19:45:00.332124 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.332132    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/834c26fa-e19d-4db2-92ea-331d49a139aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889825-rm8xg\" (UID: \"834c26fa-e19d-4db2-92ea-331d49a139aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-rm8xg"
Oct 30 19:45:00.332409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.332173    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6012f92c-1834-4e04-8d81-5a8ea39d394c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-419d0a-29889825-ntqm4\" (UID: \"6012f92c-1834-4e04-8d81-5a8ea39d394c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889825-ntqm4"
Oct 30 19:45:00.332409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.332315    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx8wz\" (UniqueName: \"kubernetes.io/projected/6012f92c-1834-4e04-8d81-5a8ea39d394c-kube-api-access-zx8wz\") pod \"maas-api-key-cleanup-e2e-worker-w1-419d0a-29889825-ntqm4\" (UID: \"6012f92c-1834-4e04-8d81-5a8ea39d394c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889825-ntqm4"
Oct 30 19:45:00.332409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.332360    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7bgl\" (UniqueName: \"kubernetes.io/projected/b62d96d6-6b2d-41e3-add9-bf09e1ca9407-kube-api-access-k7bgl\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-fdb355-29889825-nzls2\" (UID: \"b62d96d6-6b2d-41e3-add9-bf09e1ca9407\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fdb355-29889825-nzls2"
Oct 30 19:45:00.332409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.332396    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cedabb17-f317-491f-ae1b-0b5b51a80cd6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889825-bz9lm\" (UID: \"cedabb17-f317-491f-ae1b-0b5b51a80cd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889825-bz9lm"
Oct 30 19:45:00.332632 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.332456    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b62d96d6-6b2d-41e3-add9-bf09e1ca9407-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-fdb355-29889825-nzls2\" (UID: \"b62d96d6-6b2d-41e3-add9-bf09e1ca9407\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fdb355-29889825-nzls2"
Oct 30 19:45:00.332686 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.332669    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f4sx\" (UniqueName: \"kubernetes.io/projected/bf5e794c-4d01-4ab8-b211-1d2346d9b82b-kube-api-access-7f4sx\") pod \"maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889825-8j66w\" (UID: \"bf5e794c-4d01-4ab8-b211-1d2346d9b82b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889825-8j66w"
Oct 30 19:45:00.332857 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.332839    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/834c26fa-e19d-4db2-92ea-331d49a139aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889825-rm8xg\" (UID: \"834c26fa-e19d-4db2-92ea-331d49a139aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-rm8xg"
Oct 30 19:45:00.332857 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.332845    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr4gm\" (UniqueName: \"kubernetes.io/projected/cedabb17-f317-491f-ae1b-0b5b51a80cd6-kube-api-access-nr4gm\") pod \"maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889825-bz9lm\" (UID: \"cedabb17-f317-491f-ae1b-0b5b51a80cd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889825-bz9lm"
Oct 30 19:45:00.333011 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.332899    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf5e794c-4d01-4ab8-b211-1d2346d9b82b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889825-8j66w\" (UID: \"bf5e794c-4d01-4ab8-b211-1d2346d9b82b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889825-8j66w"
Oct 30 19:45:00.333214 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.333192    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b62d96d6-6b2d-41e3-add9-bf09e1ca9407-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-fdb355-29889825-nzls2\" (UID: \"b62d96d6-6b2d-41e3-add9-bf09e1ca9407\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fdb355-29889825-nzls2"
Oct 30 19:45:00.347505 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.347480    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7bgl\" (UniqueName: \"kubernetes.io/projected/b62d96d6-6b2d-41e3-add9-bf09e1ca9407-kube-api-access-k7bgl\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-fdb355-29889825-nzls2\" (UID: \"b62d96d6-6b2d-41e3-add9-bf09e1ca9407\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fdb355-29889825-nzls2"
Oct 30 19:45:00.347627 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.347567    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp42h\" (UniqueName: \"kubernetes.io/projected/834c26fa-e19d-4db2-92ea-331d49a139aa-kube-api-access-xp42h\") pod \"maas-api-key-cleanup-29889825-rm8xg\" (UID: \"834c26fa-e19d-4db2-92ea-331d49a139aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-rm8xg"
Oct 30 19:45:00.434327 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.434296    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f4sx\" (UniqueName: \"kubernetes.io/projected/bf5e794c-4d01-4ab8-b211-1d2346d9b82b-kube-api-access-7f4sx\") pod \"maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889825-8j66w\" (UID: \"bf5e794c-4d01-4ab8-b211-1d2346d9b82b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889825-8j66w"
Oct 30 19:45:00.434507 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.434391    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr4gm\" (UniqueName: \"kubernetes.io/projected/cedabb17-f317-491f-ae1b-0b5b51a80cd6-kube-api-access-nr4gm\") pod \"maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889825-bz9lm\" (UID: \"cedabb17-f317-491f-ae1b-0b5b51a80cd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889825-bz9lm"
Oct 30 19:45:00.434507 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.434422    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf5e794c-4d01-4ab8-b211-1d2346d9b82b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889825-8j66w\" (UID: \"bf5e794c-4d01-4ab8-b211-1d2346d9b82b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889825-8j66w"
Oct 30 19:45:00.434507 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.434493    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6012f92c-1834-4e04-8d81-5a8ea39d394c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-419d0a-29889825-ntqm4\" (UID: \"6012f92c-1834-4e04-8d81-5a8ea39d394c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889825-ntqm4"
Oct 30 19:45:00.434638 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.434531    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx8wz\" (UniqueName: \"kubernetes.io/projected/6012f92c-1834-4e04-8d81-5a8ea39d394c-kube-api-access-zx8wz\") pod \"maas-api-key-cleanup-e2e-worker-w1-419d0a-29889825-ntqm4\" (UID: \"6012f92c-1834-4e04-8d81-5a8ea39d394c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889825-ntqm4"
Oct 30 19:45:00.434638 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.434567    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cedabb17-f317-491f-ae1b-0b5b51a80cd6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889825-bz9lm\" (UID: \"cedabb17-f317-491f-ae1b-0b5b51a80cd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889825-bz9lm"
Oct 30 19:45:00.435183 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.435161    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf5e794c-4d01-4ab8-b211-1d2346d9b82b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889825-8j66w\" (UID: \"bf5e794c-4d01-4ab8-b211-1d2346d9b82b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889825-8j66w"
Oct 30 19:45:00.435350 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.435330    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6012f92c-1834-4e04-8d81-5a8ea39d394c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-419d0a-29889825-ntqm4\" (UID: \"6012f92c-1834-4e04-8d81-5a8ea39d394c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889825-ntqm4"
Oct 30 19:45:00.435399 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.435340    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cedabb17-f317-491f-ae1b-0b5b51a80cd6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889825-bz9lm\" (UID: \"cedabb17-f317-491f-ae1b-0b5b51a80cd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889825-bz9lm"
Oct 30 19:45:00.446391 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.446368    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f4sx\" (UniqueName: \"kubernetes.io/projected/bf5e794c-4d01-4ab8-b211-1d2346d9b82b-kube-api-access-7f4sx\") pod \"maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889825-8j66w\" (UID: \"bf5e794c-4d01-4ab8-b211-1d2346d9b82b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889825-8j66w"
Oct 30 19:45:00.446593 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.446569    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx8wz\" (UniqueName: \"kubernetes.io/projected/6012f92c-1834-4e04-8d81-5a8ea39d394c-kube-api-access-zx8wz\") pod \"maas-api-key-cleanup-e2e-worker-w1-419d0a-29889825-ntqm4\" (UID: \"6012f92c-1834-4e04-8d81-5a8ea39d394c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889825-ntqm4"
Oct 30 19:45:00.450150 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.450125    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr4gm\" (UniqueName: \"kubernetes.io/projected/cedabb17-f317-491f-ae1b-0b5b51a80cd6-kube-api-access-nr4gm\") pod \"maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889825-bz9lm\" (UID: \"cedabb17-f317-491f-ae1b-0b5b51a80cd6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889825-bz9lm"
Oct 30 19:45:00.496024 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.495934    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fdb355-29889825-nzls2"
Oct 30 19:45:00.504433 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.504410    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-rm8xg"
Oct 30 19:45:00.530636 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.530603    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889825-ntqm4"
Oct 30 19:45:00.547582 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.547133    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889825-8j66w"
Oct 30 19:45:00.573266 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.562474    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889825-bz9lm"
Oct 30 19:45:00.808682 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.808602    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-rm8xg"]
Oct 30 19:45:00.813416 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:45:00.813383    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod834c26fa_e19d_4db2_92ea_331d49a139aa.slice/crio-738368ae3931e0ba27d15d1632a886e5b89e49c46759eef9ebdc01d2f7c4a919 WatchSource:0}: Error finding container 738368ae3931e0ba27d15d1632a886e5b89e49c46759eef9ebdc01d2f7c4a919: Status 404 returned error can't find the container with id 738368ae3931e0ba27d15d1632a886e5b89e49c46759eef9ebdc01d2f7c4a919
Oct 30 19:45:00.814029 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:45:00.813996    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb62d96d6_6b2d_41e3_add9_bf09e1ca9407.slice/crio-4cfc9a3df79fa03cc75e926fc8990c347ed5e51d23d4b8fd17539a129596e6ff WatchSource:0}: Error finding container 4cfc9a3df79fa03cc75e926fc8990c347ed5e51d23d4b8fd17539a129596e6ff: Status 404 returned error can't find the container with id 4cfc9a3df79fa03cc75e926fc8990c347ed5e51d23d4b8fd17539a129596e6ff
Oct 30 19:45:00.814557 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.814533    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fdb355-29889825-nzls2"]
Oct 30 19:45:00.878188 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.878136    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889825-ntqm4"]
Oct 30 19:45:00.911041 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.911014    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fdb355-29889825-nzls2"]
Oct 30 19:45:00.923420 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.923395    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889825-8j66w"]
Oct 30 19:45:00.932298 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:45:00.932261    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf5e794c_4d01_4ab8_b211_1d2346d9b82b.slice/crio-a8547961d5e08d96301877aff534fb924cd18d075dc9e781c4503cd8a6e4cf7c WatchSource:0}: Error finding container a8547961d5e08d96301877aff534fb924cd18d075dc9e781c4503cd8a6e4cf7c: Status 404 returned error can't find the container with id a8547961d5e08d96301877aff534fb924cd18d075dc9e781c4503cd8a6e4cf7c
Oct 30 19:45:00.934770 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.934375    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889825-bz9lm"]
Oct 30 19:45:00.938919 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:45:00.938896    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcedabb17_f317_491f_ae1b_0b5b51a80cd6.slice/crio-f00fa2f20e8937fd26c0cb73a49cc372da070f06391064d961c2be0718058e8c WatchSource:0}: Error finding container f00fa2f20e8937fd26c0cb73a49cc372da070f06391064d961c2be0718058e8c: Status 404 returned error can't find the container with id f00fa2f20e8937fd26c0cb73a49cc372da070f06391064d961c2be0718058e8c
Oct 30 19:45:00.973406 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.973383    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889825-8j66w" event={"ID":"bf5e794c-4d01-4ab8-b211-1d2346d9b82b","Type":"ContainerStarted","Data":"a8547961d5e08d96301877aff534fb924cd18d075dc9e781c4503cd8a6e4cf7c"}
Oct 30 19:45:00.975188 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.975166    5043 generic.go:356] "Generic (PLEG): container finished" podID="094be84d-0dd7-47d1-87a2-e0935eb38af9" containerID="2893837ec98cdd4f8ab3c641f230f1949393ac1fa1471c7ea2762326b7160b37" exitCode=0
Oct 30 19:45:00.975332 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.975311    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3e35df-xcvqg" event={"ID":"094be84d-0dd7-47d1-87a2-e0935eb38af9","Type":"ContainerDied","Data":"2893837ec98cdd4f8ab3c641f230f1949393ac1fa1471c7ea2762326b7160b37"}
Oct 30 19:45:00.976670 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.976648    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-rm8xg" event={"ID":"834c26fa-e19d-4db2-92ea-331d49a139aa","Type":"ContainerStarted","Data":"738368ae3931e0ba27d15d1632a886e5b89e49c46759eef9ebdc01d2f7c4a919"}
Oct 30 19:45:00.977889 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.977862    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889825-bz9lm" event={"ID":"cedabb17-f317-491f-ae1b-0b5b51a80cd6","Type":"ContainerStarted","Data":"f00fa2f20e8937fd26c0cb73a49cc372da070f06391064d961c2be0718058e8c"}
Oct 30 19:45:00.979364 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.979343    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fdb355-29889825-nzls2" event={"ID":"b62d96d6-6b2d-41e3-add9-bf09e1ca9407","Type":"ContainerStarted","Data":"4cfc9a3df79fa03cc75e926fc8990c347ed5e51d23d4b8fd17539a129596e6ff"}
Oct 30 19:45:00.980837 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:00.980814    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889825-ntqm4" event={"ID":"6012f92c-1834-4e04-8d81-5a8ea39d394c","Type":"ContainerStarted","Data":"ce1c664559d22008651d18d590db6a3f102d63c8a9e47819d5dc5c14214d5c4c"}
Oct 30 19:45:01.269051 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:01.269007    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fdb355-pv6x5"]
Oct 30 19:45:01.275212 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:01.275183    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fdb355-pv6x5"]
Oct 30 19:45:01.997760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:01.997694    5043 generic.go:356] "Generic (PLEG): container finished" podID="834c26fa-e19d-4db2-92ea-331d49a139aa" containerID="fcc73cd870242673395f16c274fbf5731d408accbe2bcc5058e8b720b04cd9a6" exitCode=60
Oct 30 19:45:01.997961 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:01.997774    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-rm8xg" event={"ID":"834c26fa-e19d-4db2-92ea-331d49a139aa","Type":"ContainerDied","Data":"fcc73cd870242673395f16c274fbf5731d408accbe2bcc5058e8b720b04cd9a6"}
Oct 30 19:45:01.998103 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:01.998087    5043 scope.go:122] "RemoveContainer" containerID="fcc73cd870242673395f16c274fbf5731d408accbe2bcc5058e8b720b04cd9a6"
Oct 30 19:45:02.001556 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:02.001522    5043 generic.go:356] "Generic (PLEG): container finished" podID="cedabb17-f317-491f-ae1b-0b5b51a80cd6" containerID="455dbf9334fd49bf02eb82f98bedec5680101430b8be7c653b2bd0c3b9968b05" exitCode=60
Oct 30 19:45:02.001666 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:02.001559    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889825-bz9lm" event={"ID":"cedabb17-f317-491f-ae1b-0b5b51a80cd6","Type":"ContainerDied","Data":"455dbf9334fd49bf02eb82f98bedec5680101430b8be7c653b2bd0c3b9968b05"}
Oct 30 19:45:02.002070 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:02.001953    5043 scope.go:122] "RemoveContainer" containerID="455dbf9334fd49bf02eb82f98bedec5680101430b8be7c653b2bd0c3b9968b05"
Oct 30 19:45:02.004627 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:02.004604    5043 generic.go:356] "Generic (PLEG): container finished" podID="b62d96d6-6b2d-41e3-add9-bf09e1ca9407" containerID="c60510ad7b0c97220adaf9bc36f4a16ef46df5d9f593246c05ff9afc2cbeeee3" exitCode=6
Oct 30 19:45:02.004767 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:02.004684    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fdb355-29889825-nzls2" event={"ID":"b62d96d6-6b2d-41e3-add9-bf09e1ca9407","Type":"ContainerDied","Data":"c60510ad7b0c97220adaf9bc36f4a16ef46df5d9f593246c05ff9afc2cbeeee3"}
Oct 30 19:45:02.007101 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:02.007054    5043 generic.go:356] "Generic (PLEG): container finished" podID="6012f92c-1834-4e04-8d81-5a8ea39d394c" containerID="688d21e88e94e8b2352c909e8328ae5c87e84180ed0b2f195cc4aa903cfe7ddf" exitCode=60
Oct 30 19:45:02.007652 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:02.007597    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889825-ntqm4" event={"ID":"6012f92c-1834-4e04-8d81-5a8ea39d394c","Type":"ContainerDied","Data":"688d21e88e94e8b2352c909e8328ae5c87e84180ed0b2f195cc4aa903cfe7ddf"}
Oct 30 19:45:02.008002 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:02.007979    5043 scope.go:122] "RemoveContainer" containerID="688d21e88e94e8b2352c909e8328ae5c87e84180ed0b2f195cc4aa903cfe7ddf"
Oct 30 19:45:02.012374 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:02.012353    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889825-8j66w" event={"ID":"bf5e794c-4d01-4ab8-b211-1d2346d9b82b","Type":"ContainerStarted","Data":"4b23b3a5e0b44a3e661095842e563a5b543d2c9c0ab7ee2fcbe65ed92455d4c1"}
Oct 30 19:45:02.193427 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:02.193373    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889825-8j66w" podStartSLOduration=2.193354182 podStartE2EDuration="2.193354182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:02.192507182 +0000 UTC m=+1721.703567147" watchObservedRunningTime="2026-10-30 19:45:02.193354182 +0000 UTC m=+1721.704414155"
Oct 30 19:45:02.315923 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:02.315901    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fdb355-29889825-nzls2"
Oct 30 19:45:02.319072 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:02.319051    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3e35df-xcvqg"
Oct 30 19:45:02.385760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:02.385211    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b62d96d6-6b2d-41e3-add9-bf09e1ca9407-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b62d96d6-6b2d-41e3-add9-bf09e1ca9407-maas-api-service-ca\") pod \"b62d96d6-6b2d-41e3-add9-bf09e1ca9407\" (UID: \"b62d96d6-6b2d-41e3-add9-bf09e1ca9407\") "
Oct 30 19:45:02.385760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:02.385329    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b62d96d6-6b2d-41e3-add9-bf09e1ca9407-kube-api-access-k7bgl\" (UniqueName: \"kubernetes.io/projected/b62d96d6-6b2d-41e3-add9-bf09e1ca9407-kube-api-access-k7bgl\") pod \"b62d96d6-6b2d-41e3-add9-bf09e1ca9407\" (UID: \"b62d96d6-6b2d-41e3-add9-bf09e1ca9407\") "
Oct 30 19:45:02.385760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:02.385442    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/094be84d-0dd7-47d1-87a2-e0935eb38af9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/094be84d-0dd7-47d1-87a2-e0935eb38af9-maas-api-xks-ca\") pod \"094be84d-0dd7-47d1-87a2-e0935eb38af9\" (UID: \"094be84d-0dd7-47d1-87a2-e0935eb38af9\") "
Oct 30 19:45:02.385760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:02.385515    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/094be84d-0dd7-47d1-87a2-e0935eb38af9-kube-api-access-6dj67\" (UniqueName: \"kubernetes.io/projected/094be84d-0dd7-47d1-87a2-e0935eb38af9-kube-api-access-6dj67\") pod \"094be84d-0dd7-47d1-87a2-e0935eb38af9\" (UID: \"094be84d-0dd7-47d1-87a2-e0935eb38af9\") "
Oct 30 19:45:02.385760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:02.385589    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/094be84d-0dd7-47d1-87a2-e0935eb38af9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/094be84d-0dd7-47d1-87a2-e0935eb38af9-maas-api-openshift-ca\") pod \"094be84d-0dd7-47d1-87a2-e0935eb38af9\" (UID: \"094be84d-0dd7-47d1-87a2-e0935eb38af9\") "
Oct 30 19:45:02.388535 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:02.388498    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b62d96d6-6b2d-41e3-add9-bf09e1ca9407-maas-api-service-ca" pod "b62d96d6-6b2d-41e3-add9-bf09e1ca9407" (UID: "b62d96d6-6b2d-41e3-add9-bf09e1ca9407"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:02.390518 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:02.390490    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/094be84d-0dd7-47d1-87a2-e0935eb38af9-maas-api-openshift-ca" pod "094be84d-0dd7-47d1-87a2-e0935eb38af9" (UID: "094be84d-0dd7-47d1-87a2-e0935eb38af9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:02.405747 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:02.405682    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/094be84d-0dd7-47d1-87a2-e0935eb38af9-maas-api-xks-ca" pod "094be84d-0dd7-47d1-87a2-e0935eb38af9" (UID: "094be84d-0dd7-47d1-87a2-e0935eb38af9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:45:02.416496 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:02.416446    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b62d96d6-6b2d-41e3-add9-bf09e1ca9407-kube-api-access-k7bgl" pod "b62d96d6-6b2d-41e3-add9-bf09e1ca9407" (UID: "b62d96d6-6b2d-41e3-add9-bf09e1ca9407"). InnerVolumeSpecName "kube-api-access-k7bgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:02.459275 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:02.459243    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/094be84d-0dd7-47d1-87a2-e0935eb38af9-kube-api-access-6dj67" pod "094be84d-0dd7-47d1-87a2-e0935eb38af9" (UID: "094be84d-0dd7-47d1-87a2-e0935eb38af9"). InnerVolumeSpecName "kube-api-access-6dj67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:02.487628 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:02.487599    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dj67\" (UniqueName: \"kubernetes.io/projected/094be84d-0dd7-47d1-87a2-e0935eb38af9-kube-api-access-6dj67\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:02.487765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:02.487630    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/094be84d-0dd7-47d1-87a2-e0935eb38af9-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:02.487765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:02.487644    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b62d96d6-6b2d-41e3-add9-bf09e1ca9407-maas-api-service-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:02.487765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:02.487658    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7bgl\" (UniqueName: \"kubernetes.io/projected/b62d96d6-6b2d-41e3-add9-bf09e1ca9407-kube-api-access-k7bgl\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:02.487765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:02.487672    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/094be84d-0dd7-47d1-87a2-e0935eb38af9-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:02.647711 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:02.647623    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29414e40-8647-43d5-972c-edff8ad0e71a" path="/var/lib/kubelet/pods/29414e40-8647-43d5-972c-edff8ad0e71a/volumes"
Oct 30 19:45:03.028919 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:03.028878    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889825-ntqm4" event={"ID":"6012f92c-1834-4e04-8d81-5a8ea39d394c","Type":"ContainerStarted","Data":"bd592404fe419066983d517b72a8bbb01dbcd2786cbde78d6b4ec1c105671068"}
Oct 30 19:45:03.031412 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:03.031386    5043 generic.go:356] "Generic (PLEG): container finished" podID="bf5e794c-4d01-4ab8-b211-1d2346d9b82b" containerID="4b23b3a5e0b44a3e661095842e563a5b543d2c9c0ab7ee2fcbe65ed92455d4c1" exitCode=60
Oct 30 19:45:03.031581 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:03.031476    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889825-8j66w" event={"ID":"bf5e794c-4d01-4ab8-b211-1d2346d9b82b","Type":"ContainerDied","Data":"4b23b3a5e0b44a3e661095842e563a5b543d2c9c0ab7ee2fcbe65ed92455d4c1"}
Oct 30 19:45:03.031783 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:03.031765    5043 scope.go:122] "RemoveContainer" containerID="4b23b3a5e0b44a3e661095842e563a5b543d2c9c0ab7ee2fcbe65ed92455d4c1"
Oct 30 19:45:03.033130 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:03.033113    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3e35df-xcvqg"
Oct 30 19:45:03.033219 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:03.033202    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3e35df-xcvqg" event={"ID":"094be84d-0dd7-47d1-87a2-e0935eb38af9","Type":"ContainerDied","Data":"b431ed979dd15213a77ba97bb696acc2f3a13426b29854befbb46b487e7b9052"}
Oct 30 19:45:03.033298 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:03.033243    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b431ed979dd15213a77ba97bb696acc2f3a13426b29854befbb46b487e7b9052"
Oct 30 19:45:03.036950 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:03.036923    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-rm8xg" event={"ID":"834c26fa-e19d-4db2-92ea-331d49a139aa","Type":"ContainerStarted","Data":"0db55949da82ce7ea2496d027d2e813875203dc2cae3d6633c7e6a0c93c25674"}
Oct 30 19:45:03.039647 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:03.039625    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889825-bz9lm" event={"ID":"cedabb17-f317-491f-ae1b-0b5b51a80cd6","Type":"ContainerStarted","Data":"b6afd57d5c0be29fefe482be0f0371a2ad6524f6e5afd39ff34063ac1fdaceaa"}
Oct 30 19:45:03.041247 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:03.041227    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fdb355-29889825-nzls2" event={"ID":"b62d96d6-6b2d-41e3-add9-bf09e1ca9407","Type":"ContainerDied","Data":"4cfc9a3df79fa03cc75e926fc8990c347ed5e51d23d4b8fd17539a129596e6ff"}
Oct 30 19:45:03.041378 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:03.041254    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fdb355-29889825-nzls2"
Oct 30 19:45:03.041378 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:03.041272    5043 scope.go:122] "RemoveContainer" containerID="c60510ad7b0c97220adaf9bc36f4a16ef46df5d9f593246c05ff9afc2cbeeee3"
Oct 30 19:45:03.067975 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:03.067899    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889825-ntqm4" podStartSLOduration=3.067879671 podStartE2EDuration="3.067879671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:03.057557474 +0000 UTC m=+1722.568617440" watchObservedRunningTime="2026-10-30 19:45:03.067879671 +0000 UTC m=+1722.578939639"
Oct 30 19:45:03.093409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:03.093349    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-rm8xg" podStartSLOduration=3.093331142 podStartE2EDuration="3.093331142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:03.090025534 +0000 UTC m=+1722.601085499" watchObservedRunningTime="2026-10-30 19:45:03.093331142 +0000 UTC m=+1722.604391108"
Oct 30 19:45:03.117711 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:03.117658    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889825-bz9lm" podStartSLOduration=3.117644541 podStartE2EDuration="3.117644541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:03.117324343 +0000 UTC m=+1722.628384313" watchObservedRunningTime="2026-10-30 19:45:03.117644541 +0000 UTC m=+1722.628704535"
Oct 30 19:45:03.148975 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:03.148933    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fdb355-29889825-nzls2"]
Oct 30 19:45:03.154472 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:03.154441    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fdb355-29889825-nzls2"]
Oct 30 19:45:04.106049 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:04.102929    5043 generic.go:356] "Generic (PLEG): container finished" podID="6012f92c-1834-4e04-8d81-5a8ea39d394c" containerID="bd592404fe419066983d517b72a8bbb01dbcd2786cbde78d6b4ec1c105671068" exitCode=60
Oct 30 19:45:04.106049 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:04.103140    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889825-ntqm4" event={"ID":"6012f92c-1834-4e04-8d81-5a8ea39d394c","Type":"ContainerDied","Data":"bd592404fe419066983d517b72a8bbb01dbcd2786cbde78d6b4ec1c105671068"}
Oct 30 19:45:04.106049 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:04.103201    5043 scope.go:122] "RemoveContainer" containerID="688d21e88e94e8b2352c909e8328ae5c87e84180ed0b2f195cc4aa903cfe7ddf"
Oct 30 19:45:04.106049 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:04.103664    5043 scope.go:122] "RemoveContainer" containerID="bd592404fe419066983d517b72a8bbb01dbcd2786cbde78d6b4ec1c105671068"
Oct 30 19:45:04.106049 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:45:04.104101    5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-419d0a-29889825-ntqm4_odh-ai-gateway-infra(6012f92c-1834-4e04-8d81-5a8ea39d394c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889825-ntqm4" podUID="6012f92c-1834-4e04-8d81-5a8ea39d394c"
Oct 30 19:45:04.123369 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:04.123322    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889825-8j66w" event={"ID":"bf5e794c-4d01-4ab8-b211-1d2346d9b82b","Type":"ContainerStarted","Data":"ec1a1fd702d8b5a29619fd36499d43b3a6571e32e06aed7546db1d5cd57c0ca7"}
Oct 30 19:45:04.130147 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:04.127621    5043 generic.go:356] "Generic (PLEG): container finished" podID="834c26fa-e19d-4db2-92ea-331d49a139aa" containerID="0db55949da82ce7ea2496d027d2e813875203dc2cae3d6633c7e6a0c93c25674" exitCode=60
Oct 30 19:45:04.130147 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:04.127972    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-rm8xg" event={"ID":"834c26fa-e19d-4db2-92ea-331d49a139aa","Type":"ContainerDied","Data":"0db55949da82ce7ea2496d027d2e813875203dc2cae3d6633c7e6a0c93c25674"}
Oct 30 19:45:04.130147 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:04.128492    5043 scope.go:122] "RemoveContainer" containerID="0db55949da82ce7ea2496d027d2e813875203dc2cae3d6633c7e6a0c93c25674"
Oct 30 19:45:04.130147 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:45:04.128843    5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889825-rm8xg_odh-ai-gateway-infra(834c26fa-e19d-4db2-92ea-331d49a139aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-rm8xg" podUID="834c26fa-e19d-4db2-92ea-331d49a139aa"
Oct 30 19:45:04.130917 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:04.130891    5043 generic.go:356] "Generic (PLEG): container finished" podID="cedabb17-f317-491f-ae1b-0b5b51a80cd6" containerID="b6afd57d5c0be29fefe482be0f0371a2ad6524f6e5afd39ff34063ac1fdaceaa" exitCode=60
Oct 30 19:45:04.131024 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:04.131009    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889825-bz9lm" event={"ID":"cedabb17-f317-491f-ae1b-0b5b51a80cd6","Type":"ContainerDied","Data":"b6afd57d5c0be29fefe482be0f0371a2ad6524f6e5afd39ff34063ac1fdaceaa"}
Oct 30 19:45:04.131318 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:04.131299    5043 scope.go:122] "RemoveContainer" containerID="b6afd57d5c0be29fefe482be0f0371a2ad6524f6e5afd39ff34063ac1fdaceaa"
Oct 30 19:45:04.131555 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:45:04.131534    5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889825-bz9lm_odh-ai-gateway-infra(cedabb17-f317-491f-ae1b-0b5b51a80cd6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889825-bz9lm" podUID="cedabb17-f317-491f-ae1b-0b5b51a80cd6"
Oct 30 19:45:04.136061 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:04.136040    5043 scope.go:122] "RemoveContainer" containerID="fcc73cd870242673395f16c274fbf5731d408accbe2bcc5058e8b720b04cd9a6"
Oct 30 19:45:04.150784 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:04.150686    5043 scope.go:122] "RemoveContainer" containerID="455dbf9334fd49bf02eb82f98bedec5680101430b8be7c653b2bd0c3b9968b05"
Oct 30 19:45:04.646376 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:04.646338    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b62d96d6-6b2d-41e3-add9-bf09e1ca9407" path="/var/lib/kubelet/pods/b62d96d6-6b2d-41e3-add9-bf09e1ca9407/volumes"
Oct 30 19:45:05.162793 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:05.160758    5043 scope.go:122] "RemoveContainer" containerID="0db55949da82ce7ea2496d027d2e813875203dc2cae3d6633c7e6a0c93c25674"
Oct 30 19:45:05.162793 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:45:05.161582    5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889825-rm8xg_odh-ai-gateway-infra(834c26fa-e19d-4db2-92ea-331d49a139aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-rm8xg" podUID="834c26fa-e19d-4db2-92ea-331d49a139aa"
Oct 30 19:45:05.164271 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:05.163551    5043 scope.go:122] "RemoveContainer" containerID="b6afd57d5c0be29fefe482be0f0371a2ad6524f6e5afd39ff34063ac1fdaceaa"
Oct 30 19:45:05.164271 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:45:05.163895    5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889825-bz9lm_odh-ai-gateway-infra(cedabb17-f317-491f-ae1b-0b5b51a80cd6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889825-bz9lm" podUID="cedabb17-f317-491f-ae1b-0b5b51a80cd6"
Oct 30 19:45:05.166766 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:05.166709    5043 scope.go:122] "RemoveContainer" containerID="bd592404fe419066983d517b72a8bbb01dbcd2786cbde78d6b4ec1c105671068"
Oct 30 19:45:05.167075 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:45:05.167044    5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-419d0a-29889825-ntqm4_odh-ai-gateway-infra(6012f92c-1834-4e04-8d81-5a8ea39d394c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889825-ntqm4" podUID="6012f92c-1834-4e04-8d81-5a8ea39d394c"
Oct 30 19:45:05.171316 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:05.171288    5043 generic.go:356] "Generic (PLEG): container finished" podID="bf5e794c-4d01-4ab8-b211-1d2346d9b82b" containerID="ec1a1fd702d8b5a29619fd36499d43b3a6571e32e06aed7546db1d5cd57c0ca7" exitCode=60
Oct 30 19:45:05.171435 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:05.171359    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889825-8j66w" event={"ID":"bf5e794c-4d01-4ab8-b211-1d2346d9b82b","Type":"ContainerDied","Data":"ec1a1fd702d8b5a29619fd36499d43b3a6571e32e06aed7546db1d5cd57c0ca7"}
Oct 30 19:45:05.171435 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:05.171415    5043 scope.go:122] "RemoveContainer" containerID="4b23b3a5e0b44a3e661095842e563a5b543d2c9c0ab7ee2fcbe65ed92455d4c1"
Oct 30 19:45:05.171789 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:05.171769    5043 scope.go:122] "RemoveContainer" containerID="ec1a1fd702d8b5a29619fd36499d43b3a6571e32e06aed7546db1d5cd57c0ca7"
Oct 30 19:45:05.172087 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:45:05.172061    5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889825-8j66w_odh-ai-gateway-infra(bf5e794c-4d01-4ab8-b211-1d2346d9b82b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889825-8j66w" podUID="bf5e794c-4d01-4ab8-b211-1d2346d9b82b"
Oct 30 19:45:06.216758 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:06.216383    5043 scope.go:122] "RemoveContainer" containerID="ec1a1fd702d8b5a29619fd36499d43b3a6571e32e06aed7546db1d5cd57c0ca7"
Oct 30 19:45:06.217252 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:45:06.216993    5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889825-8j66w_odh-ai-gateway-infra(bf5e794c-4d01-4ab8-b211-1d2346d9b82b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889825-8j66w" podUID="bf5e794c-4d01-4ab8-b211-1d2346d9b82b"
Oct 30 19:45:07.967463 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:07.967426    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaeb877c-szszg"]
Oct 30 19:45:07.968489 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:07.968463    5043 cpu_manager.go:414] "containerMap: removing container" podUID="b62d96d6-6b2d-41e3-add9-bf09e1ca9407" containerName="cleanup"
Oct 30 19:45:07.968489 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:07.968490    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b62d96d6-6b2d-41e3-add9-bf09e1ca9407" containerName="cleanup"
Oct 30 19:45:07.969134 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:07.968519    5043 cpu_manager.go:414] "containerMap: removing container" podUID="094be84d-0dd7-47d1-87a2-e0935eb38af9" containerName="revoke-keys"
Oct 30 19:45:07.969134 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:07.968528    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="094be84d-0dd7-47d1-87a2-e0935eb38af9" containerName="revoke-keys"
Oct 30 19:45:07.969134 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:07.968718    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="b62d96d6-6b2d-41e3-add9-bf09e1ca9407" containerName="cleanup"
Oct 30 19:45:07.969134 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:07.968753    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="094be84d-0dd7-47d1-87a2-e0935eb38af9" containerName="revoke-keys"
Oct 30 19:45:07.973856 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:07.973835    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaeb877c-szszg"
Oct 30 19:45:08.000359 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:08.000329    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:45:08.016791 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:08.013939    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaeb877c-szszg"]
Oct 30 19:45:08.017148 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:08.017121    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84924f45-h5mdf"]
Oct 30 19:45:08.037752 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:08.037701    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84924f45-h5mdf"]
Oct 30 19:45:08.037909 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:08.037817    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84924f45-h5mdf"
Oct 30 19:45:08.080426 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:08.080382    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afb97d96-82b4-4de1-b175-f02025aba9ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eaeb877c-szszg\" (UID: \"afb97d96-82b4-4de1-b175-f02025aba9ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaeb877c-szszg"
Oct 30 19:45:08.080625 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:08.080477    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4db4df1f-ce94-4cde-9e4e-b623f5ab1edd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84924f45-h5mdf\" (UID: \"4db4df1f-ce94-4cde-9e4e-b623f5ab1edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84924f45-h5mdf"
Oct 30 19:45:08.080625 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:08.080570    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afb97d96-82b4-4de1-b175-f02025aba9ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eaeb877c-szszg\" (UID: \"afb97d96-82b4-4de1-b175-f02025aba9ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaeb877c-szszg"
Oct 30 19:45:08.080625 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:08.080611    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q54cf\" (UniqueName: \"kubernetes.io/projected/4db4df1f-ce94-4cde-9e4e-b623f5ab1edd-kube-api-access-q54cf\") pod \"maas-api-revoke-sub-84924f45-h5mdf\" (UID: \"4db4df1f-ce94-4cde-9e4e-b623f5ab1edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84924f45-h5mdf"
Oct 30 19:45:08.080826 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:08.080703    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkd4b\" (UniqueName: \"kubernetes.io/projected/afb97d96-82b4-4de1-b175-f02025aba9ad-kube-api-access-kkd4b\") pod \"maas-api-revoke-sub-eaeb877c-szszg\" (UID: \"afb97d96-82b4-4de1-b175-f02025aba9ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaeb877c-szszg"
Oct 30 19:45:08.080887 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:08.080846    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4db4df1f-ce94-4cde-9e4e-b623f5ab1edd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84924f45-h5mdf\" (UID: \"4db4df1f-ce94-4cde-9e4e-b623f5ab1edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84924f45-h5mdf"
Oct 30 19:45:08.181666 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:08.181622    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4db4df1f-ce94-4cde-9e4e-b623f5ab1edd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84924f45-h5mdf\" (UID: \"4db4df1f-ce94-4cde-9e4e-b623f5ab1edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84924f45-h5mdf"
Oct 30 19:45:08.181666 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:08.181662    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afb97d96-82b4-4de1-b175-f02025aba9ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eaeb877c-szszg\" (UID: \"afb97d96-82b4-4de1-b175-f02025aba9ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaeb877c-szszg"
Oct 30 19:45:08.181955 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:08.181677    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q54cf\" (UniqueName: \"kubernetes.io/projected/4db4df1f-ce94-4cde-9e4e-b623f5ab1edd-kube-api-access-q54cf\") pod \"maas-api-revoke-sub-84924f45-h5mdf\" (UID: \"4db4df1f-ce94-4cde-9e4e-b623f5ab1edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84924f45-h5mdf"
Oct 30 19:45:08.181955 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:08.181704    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkd4b\" (UniqueName: \"kubernetes.io/projected/afb97d96-82b4-4de1-b175-f02025aba9ad-kube-api-access-kkd4b\") pod \"maas-api-revoke-sub-eaeb877c-szszg\" (UID: \"afb97d96-82b4-4de1-b175-f02025aba9ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaeb877c-szszg"
Oct 30 19:45:08.181955 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:08.181835    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4db4df1f-ce94-4cde-9e4e-b623f5ab1edd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84924f45-h5mdf\" (UID: \"4db4df1f-ce94-4cde-9e4e-b623f5ab1edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84924f45-h5mdf"
Oct 30 19:45:08.181955 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:08.181910    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afb97d96-82b4-4de1-b175-f02025aba9ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eaeb877c-szszg\" (UID: \"afb97d96-82b4-4de1-b175-f02025aba9ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaeb877c-szszg"
Oct 30 19:45:08.182522 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:08.182487    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4db4df1f-ce94-4cde-9e4e-b623f5ab1edd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84924f45-h5mdf\" (UID: \"4db4df1f-ce94-4cde-9e4e-b623f5ab1edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84924f45-h5mdf"
Oct 30 19:45:08.182522 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:08.182512    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afb97d96-82b4-4de1-b175-f02025aba9ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eaeb877c-szszg\" (UID: \"afb97d96-82b4-4de1-b175-f02025aba9ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaeb877c-szszg"
Oct 30 19:45:08.184605 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:08.184577    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4db4df1f-ce94-4cde-9e4e-b623f5ab1edd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84924f45-h5mdf\" (UID: \"4db4df1f-ce94-4cde-9e4e-b623f5ab1edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84924f45-h5mdf"
Oct 30 19:45:08.184906 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:08.184888    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afb97d96-82b4-4de1-b175-f02025aba9ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eaeb877c-szszg\" (UID: \"afb97d96-82b4-4de1-b175-f02025aba9ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaeb877c-szszg"
Oct 30 19:45:08.197938 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:08.197913    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkd4b\" (UniqueName: \"kubernetes.io/projected/afb97d96-82b4-4de1-b175-f02025aba9ad-kube-api-access-kkd4b\") pod \"maas-api-revoke-sub-eaeb877c-szszg\" (UID: \"afb97d96-82b4-4de1-b175-f02025aba9ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaeb877c-szszg"
Oct 30 19:45:08.199320 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:08.199297    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q54cf\" (UniqueName: \"kubernetes.io/projected/4db4df1f-ce94-4cde-9e4e-b623f5ab1edd-kube-api-access-q54cf\") pod \"maas-api-revoke-sub-84924f45-h5mdf\" (UID: \"4db4df1f-ce94-4cde-9e4e-b623f5ab1edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84924f45-h5mdf"
Oct 30 19:45:08.312092 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:08.311939    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaeb877c-szszg"
Oct 30 19:45:08.352291 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:08.352256    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84924f45-h5mdf"
Oct 30 19:45:08.621625 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:08.621600    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaeb877c-szszg"]
Oct 30 19:45:08.624242 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:45:08.624213    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafb97d96_82b4_4de1_b175_f02025aba9ad.slice/crio-5c3863d31e057359a73706c6c1c2e0d1a027cbc1bbb5c9528f694e7d2f48fb95 WatchSource:0}: Error finding container 5c3863d31e057359a73706c6c1c2e0d1a027cbc1bbb5c9528f694e7d2f48fb95: Status 404 returned error can't find the container with id 5c3863d31e057359a73706c6c1c2e0d1a027cbc1bbb5c9528f694e7d2f48fb95
Oct 30 19:45:08.680629 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:08.680600    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84924f45-h5mdf"]
Oct 30 19:45:08.684924 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:45:08.684894    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4db4df1f_ce94_4cde_9e4e_b623f5ab1edd.slice/crio-cd48a8c62573b82a3903962aed532568186db7a5efb0b4f40e1f2d1938ec5c24 WatchSource:0}: Error finding container cd48a8c62573b82a3903962aed532568186db7a5efb0b4f40e1f2d1938ec5c24: Status 404 returned error can't find the container with id cd48a8c62573b82a3903962aed532568186db7a5efb0b4f40e1f2d1938ec5c24
Oct 30 19:45:09.160786 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:09.160148    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3e35df-7f747b7fb-gdjfs"]
Oct 30 19:45:09.224767 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:09.224719    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3e35df-7874fb7595-2v6xk"]
Oct 30 19:45:09.245050 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:09.244963    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84924f45-h5mdf" event={"ID":"4db4df1f-ce94-4cde-9e4e-b623f5ab1edd","Type":"ContainerStarted","Data":"04af014abfc908eb0679cabfaa48f5d6e79936ce0015cf73c3b1ced273b65220"}
Oct 30 19:45:09.245050 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:09.245019    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84924f45-h5mdf" event={"ID":"4db4df1f-ce94-4cde-9e4e-b623f5ab1edd","Type":"ContainerStarted","Data":"cd48a8c62573b82a3903962aed532568186db7a5efb0b4f40e1f2d1938ec5c24"}
Oct 30 19:45:09.247367 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:09.247339    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaeb877c-szszg" event={"ID":"afb97d96-82b4-4de1-b175-f02025aba9ad","Type":"ContainerStarted","Data":"b955bee388a3cca7c53db7a69f2e0dc5629cf8eb25ac2bdf73a9357504b5e64b"}
Oct 30 19:45:09.247564 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:09.247549    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaeb877c-szszg" event={"ID":"afb97d96-82b4-4de1-b175-f02025aba9ad","Type":"ContainerStarted","Data":"5c3863d31e057359a73706c6c1c2e0d1a027cbc1bbb5c9528f694e7d2f48fb95"}
Oct 30 19:45:09.317977 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:09.317921    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84924f45-h5mdf" podStartSLOduration=2.317906359 podStartE2EDuration="2.317906359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:09.314985337 +0000 UTC m=+1728.826045303" watchObservedRunningTime="2026-10-30 19:45:09.317906359 +0000 UTC m=+1728.828966324"
Oct 30 19:45:09.368848 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:09.368788    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaeb877c-szszg" podStartSLOduration=2.368770236 podStartE2EDuration="2.368770236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:09.368637864 +0000 UTC m=+1728.879697831" watchObservedRunningTime="2026-10-30 19:45:09.368770236 +0000 UTC m=+1728.879830200"
Oct 30 19:45:10.258079 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:10.258043    5043 generic.go:356] "Generic (PLEG): container finished" podID="4db4df1f-ce94-4cde-9e4e-b623f5ab1edd" containerID="04af014abfc908eb0679cabfaa48f5d6e79936ce0015cf73c3b1ced273b65220" exitCode=0
Oct 30 19:45:10.258561 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:10.258134    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84924f45-h5mdf" event={"ID":"4db4df1f-ce94-4cde-9e4e-b623f5ab1edd","Type":"ContainerDied","Data":"04af014abfc908eb0679cabfaa48f5d6e79936ce0015cf73c3b1ced273b65220"}
Oct 30 19:45:10.260208 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:10.260181    5043 generic.go:356] "Generic (PLEG): container finished" podID="afb97d96-82b4-4de1-b175-f02025aba9ad" containerID="b955bee388a3cca7c53db7a69f2e0dc5629cf8eb25ac2bdf73a9357504b5e64b" exitCode=0
Oct 30 19:45:10.260342 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:10.260279    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaeb877c-szszg" event={"ID":"afb97d96-82b4-4de1-b175-f02025aba9ad","Type":"ContainerDied","Data":"b955bee388a3cca7c53db7a69f2e0dc5629cf8eb25ac2bdf73a9357504b5e64b"}
Oct 30 19:45:11.454395 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:11.454353    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaeb877c-szszg"
Oct 30 19:45:11.519179 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:11.519149    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84924f45-h5mdf"
Oct 30 19:45:11.546808 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:11.546782    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afb97d96-82b4-4de1-b175-f02025aba9ad-kube-api-access-kkd4b\" (UniqueName: \"kubernetes.io/projected/afb97d96-82b4-4de1-b175-f02025aba9ad-kube-api-access-kkd4b\") pod \"afb97d96-82b4-4de1-b175-f02025aba9ad\" (UID: \"afb97d96-82b4-4de1-b175-f02025aba9ad\") "
Oct 30 19:45:11.546951 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:11.546842    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afb97d96-82b4-4de1-b175-f02025aba9ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afb97d96-82b4-4de1-b175-f02025aba9ad-maas-api-xks-ca\") pod \"afb97d96-82b4-4de1-b175-f02025aba9ad\" (UID: \"afb97d96-82b4-4de1-b175-f02025aba9ad\") "
Oct 30 19:45:11.546951 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:11.546868    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afb97d96-82b4-4de1-b175-f02025aba9ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afb97d96-82b4-4de1-b175-f02025aba9ad-maas-api-openshift-ca\") pod \"afb97d96-82b4-4de1-b175-f02025aba9ad\" (UID: \"afb97d96-82b4-4de1-b175-f02025aba9ad\") "
Oct 30 19:45:11.547306 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:11.547277    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afb97d96-82b4-4de1-b175-f02025aba9ad-maas-api-openshift-ca" pod "afb97d96-82b4-4de1-b175-f02025aba9ad" (UID: "afb97d96-82b4-4de1-b175-f02025aba9ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:11.547496 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:11.547478    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afb97d96-82b4-4de1-b175-f02025aba9ad-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:11.548975 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:11.548951    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afb97d96-82b4-4de1-b175-f02025aba9ad-maas-api-xks-ca" pod "afb97d96-82b4-4de1-b175-f02025aba9ad" (UID: "afb97d96-82b4-4de1-b175-f02025aba9ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:45:11.549045 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:11.549023    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afb97d96-82b4-4de1-b175-f02025aba9ad-kube-api-access-kkd4b" pod "afb97d96-82b4-4de1-b175-f02025aba9ad" (UID: "afb97d96-82b4-4de1-b175-f02025aba9ad"). InnerVolumeSpecName "kube-api-access-kkd4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:11.648316 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:11.648288    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4db4df1f-ce94-4cde-9e4e-b623f5ab1edd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4db4df1f-ce94-4cde-9e4e-b623f5ab1edd-maas-api-xks-ca\") pod \"4db4df1f-ce94-4cde-9e4e-b623f5ab1edd\" (UID: \"4db4df1f-ce94-4cde-9e4e-b623f5ab1edd\") "
Oct 30 19:45:11.648538 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:11.648355    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4db4df1f-ce94-4cde-9e4e-b623f5ab1edd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4db4df1f-ce94-4cde-9e4e-b623f5ab1edd-maas-api-openshift-ca\") pod \"4db4df1f-ce94-4cde-9e4e-b623f5ab1edd\" (UID: \"4db4df1f-ce94-4cde-9e4e-b623f5ab1edd\") "
Oct 30 19:45:11.648538 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:11.648435    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4db4df1f-ce94-4cde-9e4e-b623f5ab1edd-kube-api-access-q54cf\" (UniqueName: \"kubernetes.io/projected/4db4df1f-ce94-4cde-9e4e-b623f5ab1edd-kube-api-access-q54cf\") pod \"4db4df1f-ce94-4cde-9e4e-b623f5ab1edd\" (UID: \"4db4df1f-ce94-4cde-9e4e-b623f5ab1edd\") "
Oct 30 19:45:11.648708 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:11.648688    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkd4b\" (UniqueName: \"kubernetes.io/projected/afb97d96-82b4-4de1-b175-f02025aba9ad-kube-api-access-kkd4b\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:11.648802 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:11.648716    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afb97d96-82b4-4de1-b175-f02025aba9ad-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:11.648853 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:11.648834    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4db4df1f-ce94-4cde-9e4e-b623f5ab1edd-maas-api-openshift-ca" pod "4db4df1f-ce94-4cde-9e4e-b623f5ab1edd" (UID: "4db4df1f-ce94-4cde-9e4e-b623f5ab1edd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:11.650878 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:11.650853    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4db4df1f-ce94-4cde-9e4e-b623f5ab1edd-kube-api-access-q54cf" pod "4db4df1f-ce94-4cde-9e4e-b623f5ab1edd" (UID: "4db4df1f-ce94-4cde-9e4e-b623f5ab1edd"). InnerVolumeSpecName "kube-api-access-q54cf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:11.651003 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:11.650945    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4db4df1f-ce94-4cde-9e4e-b623f5ab1edd-maas-api-xks-ca" pod "4db4df1f-ce94-4cde-9e4e-b623f5ab1edd" (UID: "4db4df1f-ce94-4cde-9e4e-b623f5ab1edd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:45:11.750533 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:11.750498    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4db4df1f-ce94-4cde-9e4e-b623f5ab1edd-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:11.750533 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:11.750533    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4db4df1f-ce94-4cde-9e4e-b623f5ab1edd-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:11.750885 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:11.750548    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q54cf\" (UniqueName: \"kubernetes.io/projected/4db4df1f-ce94-4cde-9e4e-b623f5ab1edd-kube-api-access-q54cf\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:12.308760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:12.307601    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84924f45-h5mdf"
Oct 30 19:45:12.308760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:12.307656    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84924f45-h5mdf" event={"ID":"4db4df1f-ce94-4cde-9e4e-b623f5ab1edd","Type":"ContainerDied","Data":"cd48a8c62573b82a3903962aed532568186db7a5efb0b4f40e1f2d1938ec5c24"}
Oct 30 19:45:12.308760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:12.307795    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd48a8c62573b82a3903962aed532568186db7a5efb0b4f40e1f2d1938ec5c24"
Oct 30 19:45:12.314764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:12.310398    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaeb877c-szszg" event={"ID":"afb97d96-82b4-4de1-b175-f02025aba9ad","Type":"ContainerDied","Data":"5c3863d31e057359a73706c6c1c2e0d1a027cbc1bbb5c9528f694e7d2f48fb95"}
Oct 30 19:45:12.314764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:12.310455    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c3863d31e057359a73706c6c1c2e0d1a027cbc1bbb5c9528f694e7d2f48fb95"
Oct 30 19:45:12.314764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:12.310525    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaeb877c-szszg"
Oct 30 19:45:12.690090 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:12.690011    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-976b80-5vkst"]
Oct 30 19:45:12.690804 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:12.690784    5043 cpu_manager.go:414] "containerMap: removing container" podUID="afb97d96-82b4-4de1-b175-f02025aba9ad" containerName="revoke-keys"
Oct 30 19:45:12.690804 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:12.690805    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afb97d96-82b4-4de1-b175-f02025aba9ad" containerName="revoke-keys"
Oct 30 19:45:12.690981 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:12.690858    5043 cpu_manager.go:414] "containerMap: removing container" podUID="4db4df1f-ce94-4cde-9e4e-b623f5ab1edd" containerName="revoke-keys"
Oct 30 19:45:12.690981 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:12.690867    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4db4df1f-ce94-4cde-9e4e-b623f5ab1edd" containerName="revoke-keys"
Oct 30 19:45:12.691079 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:12.691024    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="4db4df1f-ce94-4cde-9e4e-b623f5ab1edd" containerName="revoke-keys"
Oct 30 19:45:12.691079 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:12.691046    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="afb97d96-82b4-4de1-b175-f02025aba9ad" containerName="revoke-keys"
Oct 30 19:45:12.694656 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:12.694636    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-976b80-5vkst"
Oct 30 19:45:12.701872 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:12.701851    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:45:12.758714 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:12.758680    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-976b80-5vkst"]
Oct 30 19:45:12.764239 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:12.764211    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f188d7c-d4f9-4369-8443-91fd13388873-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-976b80-5vkst\" (UID: \"5f188d7c-d4f9-4369-8443-91fd13388873\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-976b80-5vkst"
Oct 30 19:45:12.764405 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:12.764263    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7642t\" (UniqueName: \"kubernetes.io/projected/5f188d7c-d4f9-4369-8443-91fd13388873-kube-api-access-7642t\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-976b80-5vkst\" (UID: \"5f188d7c-d4f9-4369-8443-91fd13388873\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-976b80-5vkst"
Oct 30 19:45:12.764484 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:12.764418    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f188d7c-d4f9-4369-8443-91fd13388873-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-976b80-5vkst\" (UID: \"5f188d7c-d4f9-4369-8443-91fd13388873\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-976b80-5vkst"
Oct 30 19:45:12.866052 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:12.866015    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f188d7c-d4f9-4369-8443-91fd13388873-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-976b80-5vkst\" (UID: \"5f188d7c-d4f9-4369-8443-91fd13388873\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-976b80-5vkst"
Oct 30 19:45:12.866240 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:12.866081    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f188d7c-d4f9-4369-8443-91fd13388873-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-976b80-5vkst\" (UID: \"5f188d7c-d4f9-4369-8443-91fd13388873\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-976b80-5vkst"
Oct 30 19:45:12.866240 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:12.866122    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7642t\" (UniqueName: \"kubernetes.io/projected/5f188d7c-d4f9-4369-8443-91fd13388873-kube-api-access-7642t\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-976b80-5vkst\" (UID: \"5f188d7c-d4f9-4369-8443-91fd13388873\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-976b80-5vkst"
Oct 30 19:45:12.866899 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:12.866844    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f188d7c-d4f9-4369-8443-91fd13388873-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-976b80-5vkst\" (UID: \"5f188d7c-d4f9-4369-8443-91fd13388873\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-976b80-5vkst"
Oct 30 19:45:12.868630 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:12.868610    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f188d7c-d4f9-4369-8443-91fd13388873-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-976b80-5vkst\" (UID: \"5f188d7c-d4f9-4369-8443-91fd13388873\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-976b80-5vkst"
Oct 30 19:45:12.878790 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:12.878769    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7642t\" (UniqueName: \"kubernetes.io/projected/5f188d7c-d4f9-4369-8443-91fd13388873-kube-api-access-7642t\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-976b80-5vkst\" (UID: \"5f188d7c-d4f9-4369-8443-91fd13388873\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-976b80-5vkst"
Oct 30 19:45:13.009805 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:13.009758    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-976b80-5vkst"
Oct 30 19:45:13.296881 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:13.296796    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3e35df-xcvqg"]
Oct 30 19:45:13.307467 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:13.307443    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3e35df-xcvqg"]
Oct 30 19:45:13.393035 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:45:13.393002    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f188d7c_d4f9_4369_8443_91fd13388873.slice/crio-83d8222e042ca235ba13cdf9cd0ec3a6ff0b2647428ba65e96dbb61dcb877108 WatchSource:0}: Error finding container 83d8222e042ca235ba13cdf9cd0ec3a6ff0b2647428ba65e96dbb61dcb877108: Status 404 returned error can't find the container with id 83d8222e042ca235ba13cdf9cd0ec3a6ff0b2647428ba65e96dbb61dcb877108
Oct 30 19:45:13.398692 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:13.398663    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-976b80-5vkst"]
Oct 30 19:45:13.658200 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:13.658115    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-47110f/test-model-47110f-kserve-bd876684f-4x8lh" podUID="2528146c-d4d9-43aa-80a3-c77430db2bf6" containerName="main" containerID="cri-o://38c0469a824312283aec48f464e153ca524db6b6bae3840fe6e3f6a27a8eb1e9" gracePeriod=45
Oct 30 19:45:13.942945 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:13.942918    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-47110f/test-model-47110f-kserve-bd876684f-4x8lh"
Oct 30 19:45:14.094264 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:14.094232    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2528146c-d4d9-43aa-80a3-c77430db2bf6-kube-api-access-r2zqq\" (UniqueName: \"kubernetes.io/projected/2528146c-d4d9-43aa-80a3-c77430db2bf6-kube-api-access-r2zqq\") pod \"2528146c-d4d9-43aa-80a3-c77430db2bf6\" (UID: \"2528146c-d4d9-43aa-80a3-c77430db2bf6\") "
Oct 30 19:45:14.094444 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:14.094334    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2528146c-d4d9-43aa-80a3-c77430db2bf6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2528146c-d4d9-43aa-80a3-c77430db2bf6-model-cache\") pod \"2528146c-d4d9-43aa-80a3-c77430db2bf6\" (UID: \"2528146c-d4d9-43aa-80a3-c77430db2bf6\") "
Oct 30 19:45:14.094444 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:14.094357    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2528146c-d4d9-43aa-80a3-c77430db2bf6-tls-certs\" (UniqueName: \"kubernetes.io/secret/2528146c-d4d9-43aa-80a3-c77430db2bf6-tls-certs\") pod \"2528146c-d4d9-43aa-80a3-c77430db2bf6\" (UID: \"2528146c-d4d9-43aa-80a3-c77430db2bf6\") "
Oct 30 19:45:14.094444 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:14.094398    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2528146c-d4d9-43aa-80a3-c77430db2bf6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2528146c-d4d9-43aa-80a3-c77430db2bf6-dshm\") pod \"2528146c-d4d9-43aa-80a3-c77430db2bf6\" (UID: \"2528146c-d4d9-43aa-80a3-c77430db2bf6\") "
Oct 30 19:45:14.094444 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:14.094419    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2528146c-d4d9-43aa-80a3-c77430db2bf6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2528146c-d4d9-43aa-80a3-c77430db2bf6-tmp-dir\") pod \"2528146c-d4d9-43aa-80a3-c77430db2bf6\" (UID: \"2528146c-d4d9-43aa-80a3-c77430db2bf6\") "
Oct 30 19:45:14.094680 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:14.094484    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2528146c-d4d9-43aa-80a3-c77430db2bf6-home\" (UniqueName: \"kubernetes.io/empty-dir/2528146c-d4d9-43aa-80a3-c77430db2bf6-home\") pod \"2528146c-d4d9-43aa-80a3-c77430db2bf6\" (UID: \"2528146c-d4d9-43aa-80a3-c77430db2bf6\") "
Oct 30 19:45:14.094680 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:14.094645    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2528146c-d4d9-43aa-80a3-c77430db2bf6-model-cache" pod "2528146c-d4d9-43aa-80a3-c77430db2bf6" (UID: "2528146c-d4d9-43aa-80a3-c77430db2bf6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:14.094845 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:14.094713    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2528146c-d4d9-43aa-80a3-c77430db2bf6-tmp-dir" pod "2528146c-d4d9-43aa-80a3-c77430db2bf6" (UID: "2528146c-d4d9-43aa-80a3-c77430db2bf6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:14.094845 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:14.094826    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2528146c-d4d9-43aa-80a3-c77430db2bf6-home" pod "2528146c-d4d9-43aa-80a3-c77430db2bf6" (UID: "2528146c-d4d9-43aa-80a3-c77430db2bf6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:14.095029 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:14.095012    5043 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2528146c-d4d9-43aa-80a3-c77430db2bf6-model-cache\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:14.095093 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:14.095036    5043 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2528146c-d4d9-43aa-80a3-c77430db2bf6-tmp-dir\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:14.095093 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:14.095051    5043 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2528146c-d4d9-43aa-80a3-c77430db2bf6-home\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:14.096848 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:14.096818    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2528146c-d4d9-43aa-80a3-c77430db2bf6-dshm" pod "2528146c-d4d9-43aa-80a3-c77430db2bf6" (UID: "2528146c-d4d9-43aa-80a3-c77430db2bf6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:14.097152 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:14.097125    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2528146c-d4d9-43aa-80a3-c77430db2bf6-tls-certs" pod "2528146c-d4d9-43aa-80a3-c77430db2bf6" (UID: "2528146c-d4d9-43aa-80a3-c77430db2bf6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:45:14.097152 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:14.097134    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2528146c-d4d9-43aa-80a3-c77430db2bf6-kube-api-access-r2zqq" pod "2528146c-d4d9-43aa-80a3-c77430db2bf6" (UID: "2528146c-d4d9-43aa-80a3-c77430db2bf6"). InnerVolumeSpecName "kube-api-access-r2zqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:14.161266 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:14.161234    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-3e35df-7f747b7fb-gdjfs" podUID="97fbcbc0-ac3f-4957-a0f3-a6de47fb4a15" containerName="payload-processing" containerID="cri-o://10a34c67bbd27a947f82efd6f7fbb7e19dbc8e60f02b3fa2e4d566c834afbac4" gracePeriod=25
Oct 30 19:45:14.196225 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:14.196193    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2zqq\" (UniqueName: \"kubernetes.io/projected/2528146c-d4d9-43aa-80a3-c77430db2bf6-kube-api-access-r2zqq\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:14.196225 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:14.196218    5043 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2528146c-d4d9-43aa-80a3-c77430db2bf6-tls-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:14.196225 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:14.196228    5043 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2528146c-d4d9-43aa-80a3-c77430db2bf6-dshm\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:14.225416 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:14.225384    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3e35df-7874fb7595-2v6xk" podUID="a47d8311-eef7-47b2-b426-140198d456ee" containerName="payload-pre-processing" containerID="cri-o://55451d2e63dcffbe130ecce3215310f15e8f4545b6160d879fc1f0b99386b22e" gracePeriod=25
Oct 30 19:45:14.333189 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:14.332850    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3e35df-7f747b7fb-gdjfs" event={"ID":"97fbcbc0-ac3f-4957-a0f3-a6de47fb4a15","Type":"ContainerDied","Data":"10a34c67bbd27a947f82efd6f7fbb7e19dbc8e60f02b3fa2e4d566c834afbac4"}
Oct 30 19:45:14.333409 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:14.333367    5043 generic.go:356] "Generic (PLEG): container finished" podID="97fbcbc0-ac3f-4957-a0f3-a6de47fb4a15" containerID="10a34c67bbd27a947f82efd6f7fbb7e19dbc8e60f02b3fa2e4d566c834afbac4" exitCode=0
Oct 30 19:45:14.335148 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:14.335121    5043 generic.go:356] "Generic (PLEG): container finished" podID="2528146c-d4d9-43aa-80a3-c77430db2bf6" containerID="38c0469a824312283aec48f464e153ca524db6b6bae3840fe6e3f6a27a8eb1e9" exitCode=0
Oct 30 19:45:14.335440 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:14.335428    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-47110f/test-model-47110f-kserve-bd876684f-4x8lh"
Oct 30 19:45:14.342166 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:14.341875    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-47110f/test-model-47110f-kserve-bd876684f-4x8lh" event={"ID":"2528146c-d4d9-43aa-80a3-c77430db2bf6","Type":"ContainerDied","Data":"38c0469a824312283aec48f464e153ca524db6b6bae3840fe6e3f6a27a8eb1e9"}
Oct 30 19:45:14.342166 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:14.341918    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-47110f/test-model-47110f-kserve-bd876684f-4x8lh" event={"ID":"2528146c-d4d9-43aa-80a3-c77430db2bf6","Type":"ContainerDied","Data":"a43b9b0a4ed4c26cbf3fa11040dd329bd4ac7369f0bf404cea34691ec8f05d12"}
Oct 30 19:45:14.342166 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:14.341941    5043 scope.go:122] "RemoveContainer" containerID="38c0469a824312283aec48f464e153ca524db6b6bae3840fe6e3f6a27a8eb1e9"
Oct 30 19:45:14.345887 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:14.345257    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-976b80-5vkst" event={"ID":"5f188d7c-d4f9-4369-8443-91fd13388873","Type":"ContainerStarted","Data":"6850414af39877df6865796cdeb6356dbefd3b497adc0b6408addef5bc461ac6"}
Oct 30 19:45:14.345887 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:14.345311    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-976b80-5vkst" event={"ID":"5f188d7c-d4f9-4369-8443-91fd13388873","Type":"ContainerStarted","Data":"83d8222e042ca235ba13cdf9cd0ec3a6ff0b2647428ba65e96dbb61dcb877108"}
Oct 30 19:45:14.359698 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:14.359657    5043 scope.go:122] "RemoveContainer" containerID="38c0469a824312283aec48f464e153ca524db6b6bae3840fe6e3f6a27a8eb1e9"
Oct 30 19:45:14.360125 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:45:14.360062    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38c0469a824312283aec48f464e153ca524db6b6bae3840fe6e3f6a27a8eb1e9\": container with ID starting with 38c0469a824312283aec48f464e153ca524db6b6bae3840fe6e3f6a27a8eb1e9 not found: ID does not exist" containerID="38c0469a824312283aec48f464e153ca524db6b6bae3840fe6e3f6a27a8eb1e9"
Oct 30 19:45:14.360262 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:14.360124    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38c0469a824312283aec48f464e153ca524db6b6bae3840fe6e3f6a27a8eb1e9"} err="failed to get container status \"38c0469a824312283aec48f464e153ca524db6b6bae3840fe6e3f6a27a8eb1e9\": rpc error: code = NotFound desc = could not find container \"38c0469a824312283aec48f464e153ca524db6b6bae3840fe6e3f6a27a8eb1e9\": container with ID starting with 38c0469a824312283aec48f464e153ca524db6b6bae3840fe6e3f6a27a8eb1e9 not found: ID does not exist"
Oct 30 19:45:14.377416 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:14.373671    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-976b80-5vkst" podStartSLOduration=2.373654378 podStartE2EDuration="2.373654378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:14.37212731 +0000 UTC m=+1733.883187280" watchObservedRunningTime="2026-10-30 19:45:14.373654378 +0000 UTC m=+1733.884714346"
Oct 30 19:45:14.391297 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:14.391264    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-47110f/test-model-47110f-kserve-bd876684f-4x8lh"]
Oct 30 19:45:14.403149 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:14.402654    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-47110f/test-model-47110f-kserve-bd876684f-4x8lh"]
Oct 30 19:45:14.559000 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:14.558975    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3e35df-7f747b7fb-gdjfs"
Oct 30 19:45:14.646096 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:14.646062    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="094be84d-0dd7-47d1-87a2-e0935eb38af9" path="/var/lib/kubelet/pods/094be84d-0dd7-47d1-87a2-e0935eb38af9/volumes"
Oct 30 19:45:14.646592 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:14.646571    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2528146c-d4d9-43aa-80a3-c77430db2bf6" path="/var/lib/kubelet/pods/2528146c-d4d9-43aa-80a3-c77430db2bf6/volumes"
Oct 30 19:45:14.705714 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:14.705620    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97fbcbc0-ac3f-4957-a0f3-a6de47fb4a15-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97fbcbc0-ac3f-4957-a0f3-a6de47fb4a15-plugins-config-volume\") pod \"97fbcbc0-ac3f-4957-a0f3-a6de47fb4a15\" (UID: \"97fbcbc0-ac3f-4957-a0f3-a6de47fb4a15\") "
Oct 30 19:45:14.705714 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:14.705663    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97fbcbc0-ac3f-4957-a0f3-a6de47fb4a15-kube-api-access-2z9kv\" (UniqueName: \"kubernetes.io/projected/97fbcbc0-ac3f-4957-a0f3-a6de47fb4a15-kube-api-access-2z9kv\") pod \"97fbcbc0-ac3f-4957-a0f3-a6de47fb4a15\" (UID: \"97fbcbc0-ac3f-4957-a0f3-a6de47fb4a15\") "
Oct 30 19:45:14.714784 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:14.710076    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97fbcbc0-ac3f-4957-a0f3-a6de47fb4a15-plugins-config-volume" pod "97fbcbc0-ac3f-4957-a0f3-a6de47fb4a15" (UID: "97fbcbc0-ac3f-4957-a0f3-a6de47fb4a15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:14.714784 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:14.712954    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97fbcbc0-ac3f-4957-a0f3-a6de47fb4a15-kube-api-access-2z9kv" pod "97fbcbc0-ac3f-4957-a0f3-a6de47fb4a15" (UID: "97fbcbc0-ac3f-4957-a0f3-a6de47fb4a15"). InnerVolumeSpecName "kube-api-access-2z9kv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:14.808249 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:14.808212    5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97fbcbc0-ac3f-4957-a0f3-a6de47fb4a15-plugins-config-volume\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:14.808249 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:14.808244    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z9kv\" (UniqueName: \"kubernetes.io/projected/97fbcbc0-ac3f-4957-a0f3-a6de47fb4a15-kube-api-access-2z9kv\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:15.356458 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:15.356427    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3e35df-7f747b7fb-gdjfs"
Oct 30 19:45:15.356458 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:15.356433    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3e35df-7f747b7fb-gdjfs" event={"ID":"97fbcbc0-ac3f-4957-a0f3-a6de47fb4a15","Type":"ContainerDied","Data":"24e90f3b75a8d74aca4d22b03943a913f7e30621f8db1b24f9e5bf7c1b2a245e"}
Oct 30 19:45:15.357011 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:15.356536    5043 scope.go:122] "RemoveContainer" containerID="10a34c67bbd27a947f82efd6f7fbb7e19dbc8e60f02b3fa2e4d566c834afbac4"
Oct 30 19:45:15.360295 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:15.360268    5043 generic.go:356] "Generic (PLEG): container finished" podID="5f188d7c-d4f9-4369-8443-91fd13388873" containerID="6850414af39877df6865796cdeb6356dbefd3b497adc0b6408addef5bc461ac6" exitCode=0
Oct 30 19:45:15.360404 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:15.360324    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-976b80-5vkst" event={"ID":"5f188d7c-d4f9-4369-8443-91fd13388873","Type":"ContainerDied","Data":"6850414af39877df6865796cdeb6356dbefd3b497adc0b6408addef5bc461ac6"}
Oct 30 19:45:15.418938 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:15.418907    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3e35df-7f747b7fb-gdjfs"]
Oct 30 19:45:15.432930 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:15.432904    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3e35df-7f747b7fb-gdjfs"]
Oct 30 19:45:16.523378 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:16.523355    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-976b80-5vkst"
Oct 30 19:45:16.628623 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:16.628535    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f188d7c-d4f9-4369-8443-91fd13388873-kube-api-access-7642t\" (UniqueName: \"kubernetes.io/projected/5f188d7c-d4f9-4369-8443-91fd13388873-kube-api-access-7642t\") pod \"5f188d7c-d4f9-4369-8443-91fd13388873\" (UID: \"5f188d7c-d4f9-4369-8443-91fd13388873\") "
Oct 30 19:45:16.628830 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:16.628673    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f188d7c-d4f9-4369-8443-91fd13388873-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f188d7c-d4f9-4369-8443-91fd13388873-maas-api-openshift-ca\") pod \"5f188d7c-d4f9-4369-8443-91fd13388873\" (UID: \"5f188d7c-d4f9-4369-8443-91fd13388873\") "
Oct 30 19:45:16.628830 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:16.628792    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f188d7c-d4f9-4369-8443-91fd13388873-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f188d7c-d4f9-4369-8443-91fd13388873-maas-api-xks-ca\") pod \"5f188d7c-d4f9-4369-8443-91fd13388873\" (UID: \"5f188d7c-d4f9-4369-8443-91fd13388873\") "
Oct 30 19:45:16.629059 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:16.629029    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f188d7c-d4f9-4369-8443-91fd13388873-maas-api-openshift-ca" pod "5f188d7c-d4f9-4369-8443-91fd13388873" (UID: "5f188d7c-d4f9-4369-8443-91fd13388873"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:16.629330 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:16.629311    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f188d7c-d4f9-4369-8443-91fd13388873-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:16.631162 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:16.631139    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f188d7c-d4f9-4369-8443-91fd13388873-maas-api-xks-ca" pod "5f188d7c-d4f9-4369-8443-91fd13388873" (UID: "5f188d7c-d4f9-4369-8443-91fd13388873"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:45:16.631378 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:16.631358    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f188d7c-d4f9-4369-8443-91fd13388873-kube-api-access-7642t" pod "5f188d7c-d4f9-4369-8443-91fd13388873" (UID: "5f188d7c-d4f9-4369-8443-91fd13388873"). InnerVolumeSpecName "kube-api-access-7642t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:16.638846 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:16.638796    5043 scope.go:122] "RemoveContainer" containerID="0db55949da82ce7ea2496d027d2e813875203dc2cae3d6633c7e6a0c93c25674"
Oct 30 19:45:16.639550 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:16.639502    5043 scope.go:122] "RemoveContainer" containerID="bd592404fe419066983d517b72a8bbb01dbcd2786cbde78d6b4ec1c105671068"
Oct 30 19:45:16.643043 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:16.643025    5043 scope.go:122] "RemoveContainer" containerID="ec1a1fd702d8b5a29619fd36499d43b3a6571e32e06aed7546db1d5cd57c0ca7"
Oct 30 19:45:16.643197 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:16.643183    5043 scope.go:122] "RemoveContainer" containerID="b6afd57d5c0be29fefe482be0f0371a2ad6524f6e5afd39ff34063ac1fdaceaa"
Oct 30 19:45:16.655889 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:16.655577    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97fbcbc0-ac3f-4957-a0f3-a6de47fb4a15" path="/var/lib/kubelet/pods/97fbcbc0-ac3f-4957-a0f3-a6de47fb4a15/volumes"
Oct 30 19:45:16.730726 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:16.730700    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f188d7c-d4f9-4369-8443-91fd13388873-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:16.730883 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:16.730840    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7642t\" (UniqueName: \"kubernetes.io/projected/5f188d7c-d4f9-4369-8443-91fd13388873-kube-api-access-7642t\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:17.379989 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:17.379950    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-rm8xg" event={"ID":"834c26fa-e19d-4db2-92ea-331d49a139aa","Type":"ContainerStarted","Data":"b808389377f3d36138c902041d7484f8e8e2102fd18323dafd8ecfa30ed91406"}
Oct 30 19:45:17.382124 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:17.382097    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889825-bz9lm" event={"ID":"cedabb17-f317-491f-ae1b-0b5b51a80cd6","Type":"ContainerStarted","Data":"9bda0acc172f988e6b37e72fd62aad9f8eda987cad31cf01b33773871dcb9f18"}
Oct 30 19:45:17.384088 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:17.384065    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889825-ntqm4" event={"ID":"6012f92c-1834-4e04-8d81-5a8ea39d394c","Type":"ContainerStarted","Data":"8505302ace715573f287c978d707d3c4faad6c1be772f3193e83687941954d8f"}
Oct 30 19:45:17.385485 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:17.385467    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-976b80-5vkst"
Oct 30 19:45:17.385589 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:17.385510    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-976b80-5vkst" event={"ID":"5f188d7c-d4f9-4369-8443-91fd13388873","Type":"ContainerDied","Data":"83d8222e042ca235ba13cdf9cd0ec3a6ff0b2647428ba65e96dbb61dcb877108"}
Oct 30 19:45:17.385589 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:17.385561    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83d8222e042ca235ba13cdf9cd0ec3a6ff0b2647428ba65e96dbb61dcb877108"
Oct 30 19:45:17.387791 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:17.387763    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889825-8j66w" event={"ID":"bf5e794c-4d01-4ab8-b211-1d2346d9b82b","Type":"ContainerStarted","Data":"5d7bc62482fa72a7bdd85e895f1d35b659ee0be57a8214ff3f098daf88c84058"}
Oct 30 19:45:18.398541 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:18.398505    5043 generic.go:356] "Generic (PLEG): container finished" podID="834c26fa-e19d-4db2-92ea-331d49a139aa" containerID="b808389377f3d36138c902041d7484f8e8e2102fd18323dafd8ecfa30ed91406" exitCode=60
Oct 30 19:45:18.399077 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:18.398575    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-rm8xg" event={"ID":"834c26fa-e19d-4db2-92ea-331d49a139aa","Type":"ContainerDied","Data":"b808389377f3d36138c902041d7484f8e8e2102fd18323dafd8ecfa30ed91406"}
Oct 30 19:45:18.399077 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:18.398634    5043 scope.go:122] "RemoveContainer" containerID="0db55949da82ce7ea2496d027d2e813875203dc2cae3d6633c7e6a0c93c25674"
Oct 30 19:45:18.399077 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:18.399043    5043 scope.go:122] "RemoveContainer" containerID="b808389377f3d36138c902041d7484f8e8e2102fd18323dafd8ecfa30ed91406"
Oct 30 19:45:18.399405 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:45:18.399382    5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889825-rm8xg_odh-ai-gateway-infra(834c26fa-e19d-4db2-92ea-331d49a139aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-rm8xg" podUID="834c26fa-e19d-4db2-92ea-331d49a139aa"
Oct 30 19:45:18.401574 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:18.401552    5043 generic.go:356] "Generic (PLEG): container finished" podID="cedabb17-f317-491f-ae1b-0b5b51a80cd6" containerID="9bda0acc172f988e6b37e72fd62aad9f8eda987cad31cf01b33773871dcb9f18" exitCode=60
Oct 30 19:45:18.401715 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:18.401617    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889825-bz9lm" event={"ID":"cedabb17-f317-491f-ae1b-0b5b51a80cd6","Type":"ContainerDied","Data":"9bda0acc172f988e6b37e72fd62aad9f8eda987cad31cf01b33773871dcb9f18"}
Oct 30 19:45:18.402069 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:18.402050    5043 scope.go:122] "RemoveContainer" containerID="9bda0acc172f988e6b37e72fd62aad9f8eda987cad31cf01b33773871dcb9f18"
Oct 30 19:45:18.402408 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:45:18.402371    5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889825-bz9lm_odh-ai-gateway-infra(cedabb17-f317-491f-ae1b-0b5b51a80cd6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889825-bz9lm" podUID="cedabb17-f317-491f-ae1b-0b5b51a80cd6"
Oct 30 19:45:18.403826 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:18.403803    5043 generic.go:356] "Generic (PLEG): container finished" podID="6012f92c-1834-4e04-8d81-5a8ea39d394c" containerID="8505302ace715573f287c978d707d3c4faad6c1be772f3193e83687941954d8f" exitCode=60
Oct 30 19:45:18.403932 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:18.403878    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889825-ntqm4" event={"ID":"6012f92c-1834-4e04-8d81-5a8ea39d394c","Type":"ContainerDied","Data":"8505302ace715573f287c978d707d3c4faad6c1be772f3193e83687941954d8f"}
Oct 30 19:45:18.404234 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:18.404217    5043 scope.go:122] "RemoveContainer" containerID="8505302ace715573f287c978d707d3c4faad6c1be772f3193e83687941954d8f"
Oct 30 19:45:18.404525 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:45:18.404503    5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-419d0a-29889825-ntqm4_odh-ai-gateway-infra(6012f92c-1834-4e04-8d81-5a8ea39d394c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889825-ntqm4" podUID="6012f92c-1834-4e04-8d81-5a8ea39d394c"
Oct 30 19:45:18.406238 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:18.406219    5043 generic.go:356] "Generic (PLEG): container finished" podID="bf5e794c-4d01-4ab8-b211-1d2346d9b82b" containerID="5d7bc62482fa72a7bdd85e895f1d35b659ee0be57a8214ff3f098daf88c84058" exitCode=60
Oct 30 19:45:18.406317 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:18.406265    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889825-8j66w" event={"ID":"bf5e794c-4d01-4ab8-b211-1d2346d9b82b","Type":"ContainerDied","Data":"5d7bc62482fa72a7bdd85e895f1d35b659ee0be57a8214ff3f098daf88c84058"}
Oct 30 19:45:18.406574 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:18.406561    5043 scope.go:122] "RemoveContainer" containerID="5d7bc62482fa72a7bdd85e895f1d35b659ee0be57a8214ff3f098daf88c84058"
Oct 30 19:45:18.406903 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:45:18.406841    5043 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889825-8j66w_odh-ai-gateway-infra(bf5e794c-4d01-4ab8-b211-1d2346d9b82b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889825-8j66w" podUID="bf5e794c-4d01-4ab8-b211-1d2346d9b82b"
Oct 30 19:45:18.413016 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:18.412996    5043 scope.go:122] "RemoveContainer" containerID="b6afd57d5c0be29fefe482be0f0371a2ad6524f6e5afd39ff34063ac1fdaceaa"
Oct 30 19:45:18.427301 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:18.427283    5043 scope.go:122] "RemoveContainer" containerID="bd592404fe419066983d517b72a8bbb01dbcd2786cbde78d6b4ec1c105671068"
Oct 30 19:45:18.438673 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:18.438652    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-rm8xg"]
Oct 30 19:45:18.441585 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:18.441569    5043 scope.go:122] "RemoveContainer" containerID="ec1a1fd702d8b5a29619fd36499d43b3a6571e32e06aed7546db1d5cd57c0ca7"
Oct 30 19:45:18.477419 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:18.477391    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889825-ntqm4"]
Oct 30 19:45:18.511395 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:18.511367    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889825-bz9lm"]
Oct 30 19:45:18.579671 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:18.579636    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889825-8j66w"]
Oct 30 19:45:19.656715 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:19.656682    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-rm8xg"
Oct 30 19:45:19.715359 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:19.715334    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889825-bz9lm"
Oct 30 19:45:19.752379 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:19.752357    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889825-ntqm4"
Oct 30 19:45:19.755981 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:19.755952    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889825-8j66w"
Oct 30 19:45:19.781543 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:19.781516    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/834c26fa-e19d-4db2-92ea-331d49a139aa-kube-api-access-xp42h\" (UniqueName: \"kubernetes.io/projected/834c26fa-e19d-4db2-92ea-331d49a139aa-kube-api-access-xp42h\") pod \"834c26fa-e19d-4db2-92ea-331d49a139aa\" (UID: \"834c26fa-e19d-4db2-92ea-331d49a139aa\") "
Oct 30 19:45:19.781862 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:19.781840    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/834c26fa-e19d-4db2-92ea-331d49a139aa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/834c26fa-e19d-4db2-92ea-331d49a139aa-maas-api-service-ca\") pod \"834c26fa-e19d-4db2-92ea-331d49a139aa\" (UID: \"834c26fa-e19d-4db2-92ea-331d49a139aa\") "
Oct 30 19:45:19.783674 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:19.783651    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/834c26fa-e19d-4db2-92ea-331d49a139aa-maas-api-service-ca" pod "834c26fa-e19d-4db2-92ea-331d49a139aa" (UID: "834c26fa-e19d-4db2-92ea-331d49a139aa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:19.786282 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:19.786261    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/834c26fa-e19d-4db2-92ea-331d49a139aa-kube-api-access-xp42h" pod "834c26fa-e19d-4db2-92ea-331d49a139aa" (UID: "834c26fa-e19d-4db2-92ea-331d49a139aa"). InnerVolumeSpecName "kube-api-access-xp42h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:19.883481 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:19.883453    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cedabb17-f317-491f-ae1b-0b5b51a80cd6-kube-api-access-nr4gm\" (UniqueName: \"kubernetes.io/projected/cedabb17-f317-491f-ae1b-0b5b51a80cd6-kube-api-access-nr4gm\") pod \"cedabb17-f317-491f-ae1b-0b5b51a80cd6\" (UID: \"cedabb17-f317-491f-ae1b-0b5b51a80cd6\") "
Oct 30 19:45:19.883672 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:19.883630    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6012f92c-1834-4e04-8d81-5a8ea39d394c-kube-api-access-zx8wz\" (UniqueName: \"kubernetes.io/projected/6012f92c-1834-4e04-8d81-5a8ea39d394c-kube-api-access-zx8wz\") pod \"6012f92c-1834-4e04-8d81-5a8ea39d394c\" (UID: \"6012f92c-1834-4e04-8d81-5a8ea39d394c\") "
Oct 30 19:45:19.883773 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:19.883688    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cedabb17-f317-491f-ae1b-0b5b51a80cd6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cedabb17-f317-491f-ae1b-0b5b51a80cd6-maas-api-service-ca\") pod \"cedabb17-f317-491f-ae1b-0b5b51a80cd6\" (UID: \"cedabb17-f317-491f-ae1b-0b5b51a80cd6\") "
Oct 30 19:45:19.883773 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:19.883707    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf5e794c-4d01-4ab8-b211-1d2346d9b82b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf5e794c-4d01-4ab8-b211-1d2346d9b82b-maas-api-service-ca\") pod \"bf5e794c-4d01-4ab8-b211-1d2346d9b82b\" (UID: \"bf5e794c-4d01-4ab8-b211-1d2346d9b82b\") "
Oct 30 19:45:19.883773 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:19.883766    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6012f92c-1834-4e04-8d81-5a8ea39d394c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6012f92c-1834-4e04-8d81-5a8ea39d394c-maas-api-service-ca\") pod \"6012f92c-1834-4e04-8d81-5a8ea39d394c\" (UID: \"6012f92c-1834-4e04-8d81-5a8ea39d394c\") "
Oct 30 19:45:19.883948 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:19.883893    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf5e794c-4d01-4ab8-b211-1d2346d9b82b-kube-api-access-7f4sx\" (UniqueName: \"kubernetes.io/projected/bf5e794c-4d01-4ab8-b211-1d2346d9b82b-kube-api-access-7f4sx\") pod \"bf5e794c-4d01-4ab8-b211-1d2346d9b82b\" (UID: \"bf5e794c-4d01-4ab8-b211-1d2346d9b82b\") "
Oct 30 19:45:19.884164 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:19.884125    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf5e794c-4d01-4ab8-b211-1d2346d9b82b-maas-api-service-ca" pod "bf5e794c-4d01-4ab8-b211-1d2346d9b82b" (UID: "bf5e794c-4d01-4ab8-b211-1d2346d9b82b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:19.884164 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:19.884126    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cedabb17-f317-491f-ae1b-0b5b51a80cd6-maas-api-service-ca" pod "cedabb17-f317-491f-ae1b-0b5b51a80cd6" (UID: "cedabb17-f317-491f-ae1b-0b5b51a80cd6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:19.884164 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:19.884130    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6012f92c-1834-4e04-8d81-5a8ea39d394c-maas-api-service-ca" pod "6012f92c-1834-4e04-8d81-5a8ea39d394c" (UID: "6012f92c-1834-4e04-8d81-5a8ea39d394c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:19.884718 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:19.884697    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6012f92c-1834-4e04-8d81-5a8ea39d394c-maas-api-service-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:19.884847 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:19.884727    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/834c26fa-e19d-4db2-92ea-331d49a139aa-maas-api-service-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:19.884847 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:19.884779    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp42h\" (UniqueName: \"kubernetes.io/projected/834c26fa-e19d-4db2-92ea-331d49a139aa-kube-api-access-xp42h\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:19.884847 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:19.884795    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cedabb17-f317-491f-ae1b-0b5b51a80cd6-maas-api-service-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:19.884847 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:19.884811    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf5e794c-4d01-4ab8-b211-1d2346d9b82b-maas-api-service-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:19.886267 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:19.886244    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6012f92c-1834-4e04-8d81-5a8ea39d394c-kube-api-access-zx8wz" pod "6012f92c-1834-4e04-8d81-5a8ea39d394c" (UID: "6012f92c-1834-4e04-8d81-5a8ea39d394c"). InnerVolumeSpecName "kube-api-access-zx8wz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:19.886379 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:19.886364    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf5e794c-4d01-4ab8-b211-1d2346d9b82b-kube-api-access-7f4sx" pod "bf5e794c-4d01-4ab8-b211-1d2346d9b82b" (UID: "bf5e794c-4d01-4ab8-b211-1d2346d9b82b"). InnerVolumeSpecName "kube-api-access-7f4sx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:19.886562 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:19.886545    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cedabb17-f317-491f-ae1b-0b5b51a80cd6-kube-api-access-nr4gm" pod "cedabb17-f317-491f-ae1b-0b5b51a80cd6" (UID: "cedabb17-f317-491f-ae1b-0b5b51a80cd6"). InnerVolumeSpecName "kube-api-access-nr4gm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:19.986577 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:19.986482    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx8wz\" (UniqueName: \"kubernetes.io/projected/6012f92c-1834-4e04-8d81-5a8ea39d394c-kube-api-access-zx8wz\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:19.986577 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:19.986519    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f4sx\" (UniqueName: \"kubernetes.io/projected/bf5e794c-4d01-4ab8-b211-1d2346d9b82b-kube-api-access-7f4sx\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:19.986577 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:19.986534    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr4gm\" (UniqueName: \"kubernetes.io/projected/cedabb17-f317-491f-ae1b-0b5b51a80cd6-kube-api-access-nr4gm\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:20.442866 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:20.442772    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-rm8xg" event={"ID":"834c26fa-e19d-4db2-92ea-331d49a139aa","Type":"ContainerDied","Data":"738368ae3931e0ba27d15d1632a886e5b89e49c46759eef9ebdc01d2f7c4a919"}
Oct 30 19:45:20.442866 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:20.442803    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-rm8xg"
Oct 30 19:45:20.442866 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:20.442841    5043 scope.go:122] "RemoveContainer" containerID="b808389377f3d36138c902041d7484f8e8e2102fd18323dafd8ecfa30ed91406"
Oct 30 19:45:20.444578 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:20.444555    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889825-bz9lm"
Oct 30 19:45:20.444715 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:20.444579    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889825-bz9lm" event={"ID":"cedabb17-f317-491f-ae1b-0b5b51a80cd6","Type":"ContainerDied","Data":"f00fa2f20e8937fd26c0cb73a49cc372da070f06391064d961c2be0718058e8c"}
Oct 30 19:45:20.446471 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:20.446452    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889825-ntqm4"
Oct 30 19:45:20.446471 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:20.446460    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889825-ntqm4" event={"ID":"6012f92c-1834-4e04-8d81-5a8ea39d394c","Type":"ContainerDied","Data":"ce1c664559d22008651d18d590db6a3f102d63c8a9e47819d5dc5c14214d5c4c"}
Oct 30 19:45:20.451309 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:20.451275    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889825-8j66w" event={"ID":"bf5e794c-4d01-4ab8-b211-1d2346d9b82b","Type":"ContainerDied","Data":"a8547961d5e08d96301877aff534fb924cd18d075dc9e781c4503cd8a6e4cf7c"}
Oct 30 19:45:20.451461 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:20.451443    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889825-8j66w"
Oct 30 19:45:20.462984 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:20.459342    5043 scope.go:122] "RemoveContainer" containerID="9bda0acc172f988e6b37e72fd62aad9f8eda987cad31cf01b33773871dcb9f18"
Oct 30 19:45:20.477893 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:20.477872    5043 scope.go:122] "RemoveContainer" containerID="8505302ace715573f287c978d707d3c4faad6c1be772f3193e83687941954d8f"
Oct 30 19:45:20.486328 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:20.486302    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889825-ntqm4"]
Oct 30 19:45:20.494914 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:20.494884    5043 scope.go:122] "RemoveContainer" containerID="5d7bc62482fa72a7bdd85e895f1d35b659ee0be57a8214ff3f098daf88c84058"
Oct 30 19:45:20.498408 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:20.498388    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-419d0a-29889825-ntqm4"]
Oct 30 19:45:20.511513 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:20.511491    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889825-bz9lm"]
Oct 30 19:45:20.517850 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:20.517818    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad96a4-29889825-bz9lm"]
Oct 30 19:45:20.541768 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:20.541722    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-rm8xg"]
Oct 30 19:45:20.549993 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:20.549967    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-rm8xg"]
Oct 30 19:45:20.568598 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:20.568573    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889825-8j66w"]
Oct 30 19:45:20.575025 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:20.575003    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ce69f3-29889825-8j66w"]
Oct 30 19:45:20.644903 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:20.644873    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6012f92c-1834-4e04-8d81-5a8ea39d394c" path="/var/lib/kubelet/pods/6012f92c-1834-4e04-8d81-5a8ea39d394c/volumes"
Oct 30 19:45:20.645381 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:20.645359    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="834c26fa-e19d-4db2-92ea-331d49a139aa" path="/var/lib/kubelet/pods/834c26fa-e19d-4db2-92ea-331d49a139aa/volumes"
Oct 30 19:45:20.645854 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:20.645834    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf5e794c-4d01-4ab8-b211-1d2346d9b82b" path="/var/lib/kubelet/pods/bf5e794c-4d01-4ab8-b211-1d2346d9b82b/volumes"
Oct 30 19:45:20.646607 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:20.646587    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cedabb17-f317-491f-ae1b-0b5b51a80cd6" path="/var/lib/kubelet/pods/cedabb17-f317-491f-ae1b-0b5b51a80cd6/volumes"
Oct 30 19:45:21.226957 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:21.226918    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3e35df-7874fb7595-2v6xk" podUID="a47d8311-eef7-47b2-b426-140198d456ee" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:45:22.870581 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:22.870543    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-976b80-646cd76644-rlkvm"]
Oct 30 19:45:24.587939 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:24.587879    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq"]
Oct 30 19:45:24.588332 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:24.588137    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq" podUID="3c89df4b-9274-4375-8ff8-47763bbf9182" containerName="istio-proxy" containerID="cri-o://713483e9da389aa486b8e73e69d2718461ac01515c9c1c6aa61da6873afbc446" gracePeriod=30
Oct 30 19:45:24.841720 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:24.841633    5043 scope.go:122] "RemoveContainer" containerID="c635515c328b8d3f93e621ab8e761c5c2f5316c99e9fc4184214454186943912"
Oct 30 19:45:24.854132 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:24.854103    5043 scope.go:122] "RemoveContainer" containerID="badd5120fa261a21202391d4f10736dc2aceca3ee86f9d9428525add2140f4fd"
Oct 30 19:45:24.866885 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:24.866861    5043 scope.go:122] "RemoveContainer" containerID="248e4b90b3b646f84dc3786f29009e13debc8f43c524da7dbad05def5d30719e"
Oct 30 19:45:24.983638 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:45:24.983606    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:45:25.042960 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.042931    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-419d0a-7hw77"]
Oct 30 19:45:25.057003 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.056971    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-419d0a-7hw77"]
Oct 30 19:45:25.180242 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.180160    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-47110f-mc9ph"]
Oct 30 19:45:25.180678 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.180665    5043 cpu_manager.go:414] "containerMap: removing container" podUID="bf5e794c-4d01-4ab8-b211-1d2346d9b82b" containerName="cleanup"
Oct 30 19:45:25.180722 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.180680    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf5e794c-4d01-4ab8-b211-1d2346d9b82b" containerName="cleanup"
Oct 30 19:45:25.180722 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.180699    5043 cpu_manager.go:414] "containerMap: removing container" podUID="bf5e794c-4d01-4ab8-b211-1d2346d9b82b" containerName="cleanup"
Oct 30 19:45:25.180722 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.180704    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf5e794c-4d01-4ab8-b211-1d2346d9b82b" containerName="cleanup"
Oct 30 19:45:25.180722 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.180714    5043 cpu_manager.go:414] "containerMap: removing container" podUID="6012f92c-1834-4e04-8d81-5a8ea39d394c" containerName="cleanup"
Oct 30 19:45:25.180722 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.180719    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6012f92c-1834-4e04-8d81-5a8ea39d394c" containerName="cleanup"
Oct 30 19:45:25.180924 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.180725    5043 cpu_manager.go:414] "containerMap: removing container" podUID="2528146c-d4d9-43aa-80a3-c77430db2bf6" containerName="main"
Oct 30 19:45:25.180924 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.180730    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2528146c-d4d9-43aa-80a3-c77430db2bf6" containerName="main"
Oct 30 19:45:25.180924 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.180753    5043 cpu_manager.go:414] "containerMap: removing container" podUID="834c26fa-e19d-4db2-92ea-331d49a139aa" containerName="cleanup"
Oct 30 19:45:25.180924 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.180760    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="834c26fa-e19d-4db2-92ea-331d49a139aa" containerName="cleanup"
Oct 30 19:45:25.180924 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.180768    5043 cpu_manager.go:414] "containerMap: removing container" podUID="6012f92c-1834-4e04-8d81-5a8ea39d394c" containerName="cleanup"
Oct 30 19:45:25.180924 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.180774    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6012f92c-1834-4e04-8d81-5a8ea39d394c" containerName="cleanup"
Oct 30 19:45:25.180924 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.180788    5043 cpu_manager.go:414] "containerMap: removing container" podUID="97fbcbc0-ac3f-4957-a0f3-a6de47fb4a15" containerName="payload-processing"
Oct 30 19:45:25.180924 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.180795    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97fbcbc0-ac3f-4957-a0f3-a6de47fb4a15" containerName="payload-processing"
Oct 30 19:45:25.180924 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.180803    5043 cpu_manager.go:414] "containerMap: removing container" podUID="bf5e794c-4d01-4ab8-b211-1d2346d9b82b" containerName="cleanup"
Oct 30 19:45:25.180924 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.180807    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf5e794c-4d01-4ab8-b211-1d2346d9b82b" containerName="cleanup"
Oct 30 19:45:25.180924 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.180824    5043 cpu_manager.go:414] "containerMap: removing container" podUID="834c26fa-e19d-4db2-92ea-331d49a139aa" containerName="cleanup"
Oct 30 19:45:25.180924 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.180828    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="834c26fa-e19d-4db2-92ea-331d49a139aa" containerName="cleanup"
Oct 30 19:45:25.180924 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.180834    5043 cpu_manager.go:414] "containerMap: removing container" podUID="cedabb17-f317-491f-ae1b-0b5b51a80cd6" containerName="cleanup"
Oct 30 19:45:25.180924 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.180839    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cedabb17-f317-491f-ae1b-0b5b51a80cd6" containerName="cleanup"
Oct 30 19:45:25.180924 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.180847    5043 cpu_manager.go:414] "containerMap: removing container" podUID="cedabb17-f317-491f-ae1b-0b5b51a80cd6" containerName="cleanup"
Oct 30 19:45:25.180924 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.180851    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cedabb17-f317-491f-ae1b-0b5b51a80cd6" containerName="cleanup"
Oct 30 19:45:25.180924 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.180873    5043 cpu_manager.go:414] "containerMap: removing container" podUID="5f188d7c-d4f9-4369-8443-91fd13388873" containerName="revoke-keys"
Oct 30 19:45:25.180924 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.180877    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f188d7c-d4f9-4369-8443-91fd13388873" containerName="revoke-keys"
Oct 30 19:45:25.181451 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.180959    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="834c26fa-e19d-4db2-92ea-331d49a139aa" containerName="cleanup"
Oct 30 19:45:25.181451 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.180965    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f188d7c-d4f9-4369-8443-91fd13388873" containerName="revoke-keys"
Oct 30 19:45:25.181451 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.180972    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf5e794c-4d01-4ab8-b211-1d2346d9b82b" containerName="cleanup"
Oct 30 19:45:25.181451 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.180979    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf5e794c-4d01-4ab8-b211-1d2346d9b82b" containerName="cleanup"
Oct 30 19:45:25.181451 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.180985    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="6012f92c-1834-4e04-8d81-5a8ea39d394c" containerName="cleanup"
Oct 30 19:45:25.181451 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.180990    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="2528146c-d4d9-43aa-80a3-c77430db2bf6" containerName="main"
Oct 30 19:45:25.181451 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.180996    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="834c26fa-e19d-4db2-92ea-331d49a139aa" containerName="cleanup"
Oct 30 19:45:25.181451 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.181001    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="6012f92c-1834-4e04-8d81-5a8ea39d394c" containerName="cleanup"
Oct 30 19:45:25.181451 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.181006    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="97fbcbc0-ac3f-4957-a0f3-a6de47fb4a15" containerName="payload-processing"
Oct 30 19:45:25.181451 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.181013    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="cedabb17-f317-491f-ae1b-0b5b51a80cd6" containerName="cleanup"
Oct 30 19:45:25.181451 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.181017    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="cedabb17-f317-491f-ae1b-0b5b51a80cd6" containerName="cleanup"
Oct 30 19:45:25.181451 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.181024    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="cedabb17-f317-491f-ae1b-0b5b51a80cd6" containerName="cleanup"
Oct 30 19:45:25.187931 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.187913    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-47110f-mc9ph"
Oct 30 19:45:25.201816 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.201785    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-47110f-mc9ph"]
Oct 30 19:45:25.205134 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.205091    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:45:25.205332 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.205312    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:45:25.205388 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.205318    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:45:25.205388 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.205325    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:45:25.270662 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.270635    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zs9h\" (UniqueName: \"kubernetes.io/projected/f6a8f7ef-e408-43d3-8816-9483b8754819-kube-api-access-2zs9h\") pod \"maas-api-revoke-keys-e2e-inf-b-47110f-mc9ph\" (UID: \"f6a8f7ef-e408-43d3-8816-9483b8754819\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-47110f-mc9ph"
Oct 30 19:45:25.270855 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.270711    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a8f7ef-e408-43d3-8816-9483b8754819-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-47110f-mc9ph\" (UID: \"f6a8f7ef-e408-43d3-8816-9483b8754819\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-47110f-mc9ph"
Oct 30 19:45:25.270907 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.270858    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a8f7ef-e408-43d3-8816-9483b8754819-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-47110f-mc9ph\" (UID: \"f6a8f7ef-e408-43d3-8816-9483b8754819\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-47110f-mc9ph"
Oct 30 19:45:25.371725 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.371687    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a8f7ef-e408-43d3-8816-9483b8754819-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-47110f-mc9ph\" (UID: \"f6a8f7ef-e408-43d3-8816-9483b8754819\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-47110f-mc9ph"
Oct 30 19:45:25.371926 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.371815    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zs9h\" (UniqueName: \"kubernetes.io/projected/f6a8f7ef-e408-43d3-8816-9483b8754819-kube-api-access-2zs9h\") pod \"maas-api-revoke-keys-e2e-inf-b-47110f-mc9ph\" (UID: \"f6a8f7ef-e408-43d3-8816-9483b8754819\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-47110f-mc9ph"
Oct 30 19:45:25.371990 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.371940    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a8f7ef-e408-43d3-8816-9483b8754819-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-47110f-mc9ph\" (UID: \"f6a8f7ef-e408-43d3-8816-9483b8754819\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-47110f-mc9ph"
Oct 30 19:45:25.372578 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.372542    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a8f7ef-e408-43d3-8816-9483b8754819-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-47110f-mc9ph\" (UID: \"f6a8f7ef-e408-43d3-8816-9483b8754819\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-47110f-mc9ph"
Oct 30 19:45:25.374252 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.374229    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a8f7ef-e408-43d3-8816-9483b8754819-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-47110f-mc9ph\" (UID: \"f6a8f7ef-e408-43d3-8816-9483b8754819\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-47110f-mc9ph"
Oct 30 19:45:25.381292 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.381268    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zs9h\" (UniqueName: \"kubernetes.io/projected/f6a8f7ef-e408-43d3-8816-9483b8754819-kube-api-access-2zs9h\") pod \"maas-api-revoke-keys-e2e-inf-b-47110f-mc9ph\" (UID: \"f6a8f7ef-e408-43d3-8816-9483b8754819\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-47110f-mc9ph"
Oct 30 19:45:25.502590 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.502561    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-47110f-mc9ph"
Oct 30 19:45:25.795152 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:25.795119    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-47110f-mc9ph"]
Oct 30 19:45:26.515111 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:26.515071    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-47110f-mc9ph" event={"ID":"f6a8f7ef-e408-43d3-8816-9483b8754819","Type":"ContainerStarted","Data":"0877106a08d4de32c0887a0f3cc22740074c951749a108878c293cfa5226f969"}
Oct 30 19:45:26.515301 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:26.515139    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-47110f-mc9ph" event={"ID":"f6a8f7ef-e408-43d3-8816-9483b8754819","Type":"ContainerStarted","Data":"2a07bd32f06e3426b79c21002abd6c8bd4f776d19568a1c8bd260cbf3a90fce9"}
Oct 30 19:45:26.553388 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:26.553333    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-47110f-mc9ph" podStartSLOduration=1.5533188820000001 podStartE2EDuration="1.553318882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:26.54847345 +0000 UTC m=+1746.059533414" watchObservedRunningTime="2026-10-30 19:45:26.553318882 +0000 UTC m=+1746.064378846"
Oct 30 19:45:26.646426 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:26.646396    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bf48130-92b3-4703-8a2a-4bc57504a3c9" path="/var/lib/kubelet/pods/0bf48130-92b3-4703-8a2a-4bc57504a3c9/volumes"
Oct 30 19:45:27.531658 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:27.531620    5043 generic.go:356] "Generic (PLEG): container finished" podID="f6a8f7ef-e408-43d3-8816-9483b8754819" containerID="0877106a08d4de32c0887a0f3cc22740074c951749a108878c293cfa5226f969" exitCode=0
Oct 30 19:45:27.532053 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:27.531710    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-47110f-mc9ph" event={"ID":"f6a8f7ef-e408-43d3-8816-9483b8754819","Type":"ContainerDied","Data":"0877106a08d4de32c0887a0f3cc22740074c951749a108878c293cfa5226f969"}
Oct 30 19:45:27.871886 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:27.871800    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-976b80-646cd76644-rlkvm" podUID="741a0acd-9951-4c23-abf6-1c9c22e15366" containerName="payload-processing" containerID="cri-o://eac0421f5c972cb7ae74a069dc9cbe4fc272a290e5d4aeab35fcf25eb8ee4762" gracePeriod=25
Oct 30 19:45:27.922579 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:27.922547    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-976b80-5vkst"]
Oct 30 19:45:27.928198 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:27.928172    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-976b80-5vkst"]
Oct 30 19:45:28.162724 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:28.162699    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-976b80-646cd76644-rlkvm"
Oct 30 19:45:28.208450 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:28.208411    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/741a0acd-9951-4c23-abf6-1c9c22e15366-kube-api-access-w6vhj\" (UniqueName: \"kubernetes.io/projected/741a0acd-9951-4c23-abf6-1c9c22e15366-kube-api-access-w6vhj\") pod \"741a0acd-9951-4c23-abf6-1c9c22e15366\" (UID: \"741a0acd-9951-4c23-abf6-1c9c22e15366\") "
Oct 30 19:45:28.208637 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:28.208580    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/741a0acd-9951-4c23-abf6-1c9c22e15366-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/741a0acd-9951-4c23-abf6-1c9c22e15366-plugins-config-volume\") pod \"741a0acd-9951-4c23-abf6-1c9c22e15366\" (UID: \"741a0acd-9951-4c23-abf6-1c9c22e15366\") "
Oct 30 19:45:28.209019 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:28.208991    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/741a0acd-9951-4c23-abf6-1c9c22e15366-plugins-config-volume" pod "741a0acd-9951-4c23-abf6-1c9c22e15366" (UID: "741a0acd-9951-4c23-abf6-1c9c22e15366"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:28.210709 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:28.210679    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/741a0acd-9951-4c23-abf6-1c9c22e15366-kube-api-access-w6vhj" pod "741a0acd-9951-4c23-abf6-1c9c22e15366" (UID: "741a0acd-9951-4c23-abf6-1c9c22e15366"). InnerVolumeSpecName "kube-api-access-w6vhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:28.310205 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:28.310170    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6vhj\" (UniqueName: \"kubernetes.io/projected/741a0acd-9951-4c23-abf6-1c9c22e15366-kube-api-access-w6vhj\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:28.310205 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:28.310200    5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/741a0acd-9951-4c23-abf6-1c9c22e15366-plugins-config-volume\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:28.540652 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:28.540616    5043 generic.go:356] "Generic (PLEG): container finished" podID="741a0acd-9951-4c23-abf6-1c9c22e15366" containerID="eac0421f5c972cb7ae74a069dc9cbe4fc272a290e5d4aeab35fcf25eb8ee4762" exitCode=0
Oct 30 19:45:28.541140 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:28.540666    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-976b80-646cd76644-rlkvm"
Oct 30 19:45:28.541140 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:28.540704    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-976b80-646cd76644-rlkvm" event={"ID":"741a0acd-9951-4c23-abf6-1c9c22e15366","Type":"ContainerDied","Data":"eac0421f5c972cb7ae74a069dc9cbe4fc272a290e5d4aeab35fcf25eb8ee4762"}
Oct 30 19:45:28.541140 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:28.540781    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-976b80-646cd76644-rlkvm" event={"ID":"741a0acd-9951-4c23-abf6-1c9c22e15366","Type":"ContainerDied","Data":"5f00057a07f5622b31f5a86ae626173186927222c1203ae10415a743cb3f1448"}
Oct 30 19:45:28.541140 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:28.540807    5043 scope.go:122] "RemoveContainer" containerID="eac0421f5c972cb7ae74a069dc9cbe4fc272a290e5d4aeab35fcf25eb8ee4762"
Oct 30 19:45:28.553008 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:28.552982    5043 scope.go:122] "RemoveContainer" containerID="eac0421f5c972cb7ae74a069dc9cbe4fc272a290e5d4aeab35fcf25eb8ee4762"
Oct 30 19:45:28.553352 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:45:28.553322    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eac0421f5c972cb7ae74a069dc9cbe4fc272a290e5d4aeab35fcf25eb8ee4762\": container with ID starting with eac0421f5c972cb7ae74a069dc9cbe4fc272a290e5d4aeab35fcf25eb8ee4762 not found: ID does not exist" containerID="eac0421f5c972cb7ae74a069dc9cbe4fc272a290e5d4aeab35fcf25eb8ee4762"
Oct 30 19:45:28.553447 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:28.553364    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eac0421f5c972cb7ae74a069dc9cbe4fc272a290e5d4aeab35fcf25eb8ee4762"} err="failed to get container status \"eac0421f5c972cb7ae74a069dc9cbe4fc272a290e5d4aeab35fcf25eb8ee4762\": rpc error: code = NotFound desc = could not find container \"eac0421f5c972cb7ae74a069dc9cbe4fc272a290e5d4aeab35fcf25eb8ee4762\": container with ID starting with eac0421f5c972cb7ae74a069dc9cbe4fc272a290e5d4aeab35fcf25eb8ee4762 not found: ID does not exist"
Oct 30 19:45:28.574820 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:28.574785    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-976b80-646cd76644-rlkvm"]
Oct 30 19:45:28.588010 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:28.587981    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-976b80-646cd76644-rlkvm"]
Oct 30 19:45:28.648607 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:28.648576    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f188d7c-d4f9-4369-8443-91fd13388873" path="/var/lib/kubelet/pods/5f188d7c-d4f9-4369-8443-91fd13388873/volumes"
Oct 30 19:45:28.649013 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:28.648997    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="741a0acd-9951-4c23-abf6-1c9c22e15366" path="/var/lib/kubelet/pods/741a0acd-9951-4c23-abf6-1c9c22e15366/volumes"
Oct 30 19:45:28.707448 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:28.707421    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-47110f-mc9ph"
Oct 30 19:45:28.815924 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:28.815838    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6a8f7ef-e408-43d3-8816-9483b8754819-kube-api-access-2zs9h\" (UniqueName: \"kubernetes.io/projected/f6a8f7ef-e408-43d3-8816-9483b8754819-kube-api-access-2zs9h\") pod \"f6a8f7ef-e408-43d3-8816-9483b8754819\" (UID: \"f6a8f7ef-e408-43d3-8816-9483b8754819\") "
Oct 30 19:45:28.816084 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:28.816026    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6a8f7ef-e408-43d3-8816-9483b8754819-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a8f7ef-e408-43d3-8816-9483b8754819-maas-api-xks-ca\") pod \"f6a8f7ef-e408-43d3-8816-9483b8754819\" (UID: \"f6a8f7ef-e408-43d3-8816-9483b8754819\") "
Oct 30 19:45:28.816084 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:28.816072    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6a8f7ef-e408-43d3-8816-9483b8754819-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a8f7ef-e408-43d3-8816-9483b8754819-maas-api-openshift-ca\") pod \"f6a8f7ef-e408-43d3-8816-9483b8754819\" (UID: \"f6a8f7ef-e408-43d3-8816-9483b8754819\") "
Oct 30 19:45:28.816483 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:28.816454    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6a8f7ef-e408-43d3-8816-9483b8754819-maas-api-openshift-ca" pod "f6a8f7ef-e408-43d3-8816-9483b8754819" (UID: "f6a8f7ef-e408-43d3-8816-9483b8754819"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:28.816601 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:28.816550    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a8f7ef-e408-43d3-8816-9483b8754819-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:28.818175 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:28.818148    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6a8f7ef-e408-43d3-8816-9483b8754819-kube-api-access-2zs9h" pod "f6a8f7ef-e408-43d3-8816-9483b8754819" (UID: "f6a8f7ef-e408-43d3-8816-9483b8754819"). InnerVolumeSpecName "kube-api-access-2zs9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:28.818357 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:28.818343    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6a8f7ef-e408-43d3-8816-9483b8754819-maas-api-xks-ca" pod "f6a8f7ef-e408-43d3-8816-9483b8754819" (UID: "f6a8f7ef-e408-43d3-8816-9483b8754819"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:45:28.917583 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:28.917551    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zs9h\" (UniqueName: \"kubernetes.io/projected/f6a8f7ef-e408-43d3-8816-9483b8754819-kube-api-access-2zs9h\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:28.917583 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:28.917577    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a8f7ef-e408-43d3-8816-9483b8754819-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:29.550105 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:29.550078    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-47110f-mc9ph"
Oct 30 19:45:29.550503 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:29.550111    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-47110f-mc9ph" event={"ID":"f6a8f7ef-e408-43d3-8816-9483b8754819","Type":"ContainerDied","Data":"2a07bd32f06e3426b79c21002abd6c8bd4f776d19568a1c8bd260cbf3a90fce9"}
Oct 30 19:45:29.550503 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:29.550167    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a07bd32f06e3426b79c21002abd6c8bd4f776d19568a1c8bd260cbf3a90fce9"
Oct 30 19:45:29.908137 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:29.908110    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq"
Oct 30 19:45:29.930363 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:29.930336    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c89df4b-9274-4375-8ff8-47763bbf9182-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c89df4b-9274-4375-8ff8-47763bbf9182-istio-podinfo\") pod \"3c89df4b-9274-4375-8ff8-47763bbf9182\" (UID: \"3c89df4b-9274-4375-8ff8-47763bbf9182\") "
Oct 30 19:45:29.930554 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:29.930385    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c89df4b-9274-4375-8ff8-47763bbf9182-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c89df4b-9274-4375-8ff8-47763bbf9182-credential-socket\") pod \"3c89df4b-9274-4375-8ff8-47763bbf9182\" (UID: \"3c89df4b-9274-4375-8ff8-47763bbf9182\") "
Oct 30 19:45:29.930554 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:29.930432    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c89df4b-9274-4375-8ff8-47763bbf9182-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c89df4b-9274-4375-8ff8-47763bbf9182-workload-socket\") pod \"3c89df4b-9274-4375-8ff8-47763bbf9182\" (UID: \"3c89df4b-9274-4375-8ff8-47763bbf9182\") "
Oct 30 19:45:29.930554 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:29.930474    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c89df4b-9274-4375-8ff8-47763bbf9182-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c89df4b-9274-4375-8ff8-47763bbf9182-istiod-ca-cert\") pod \"3c89df4b-9274-4375-8ff8-47763bbf9182\" (UID: \"3c89df4b-9274-4375-8ff8-47763bbf9182\") "
Oct 30 19:45:29.930554 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:29.930498    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c89df4b-9274-4375-8ff8-47763bbf9182-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c89df4b-9274-4375-8ff8-47763bbf9182-istio-envoy\") pod \"3c89df4b-9274-4375-8ff8-47763bbf9182\" (UID: \"3c89df4b-9274-4375-8ff8-47763bbf9182\") "
Oct 30 19:45:29.930842 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:29.930560    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c89df4b-9274-4375-8ff8-47763bbf9182-kube-api-access-kk2sb\" (UniqueName: \"kubernetes.io/projected/3c89df4b-9274-4375-8ff8-47763bbf9182-kube-api-access-kk2sb\") pod \"3c89df4b-9274-4375-8ff8-47763bbf9182\" (UID: \"3c89df4b-9274-4375-8ff8-47763bbf9182\") "
Oct 30 19:45:29.930842 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:29.930589    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c89df4b-9274-4375-8ff8-47763bbf9182-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c89df4b-9274-4375-8ff8-47763bbf9182-istio-data\") pod \"3c89df4b-9274-4375-8ff8-47763bbf9182\" (UID: \"3c89df4b-9274-4375-8ff8-47763bbf9182\") "
Oct 30 19:45:29.930842 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:29.930671    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c89df4b-9274-4375-8ff8-47763bbf9182-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c89df4b-9274-4375-8ff8-47763bbf9182-workload-certs\") pod \"3c89df4b-9274-4375-8ff8-47763bbf9182\" (UID: \"3c89df4b-9274-4375-8ff8-47763bbf9182\") "
Oct 30 19:45:29.930842 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:29.930712    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c89df4b-9274-4375-8ff8-47763bbf9182-istio-token\" (UniqueName: \"kubernetes.io/projected/3c89df4b-9274-4375-8ff8-47763bbf9182-istio-token\") pod \"3c89df4b-9274-4375-8ff8-47763bbf9182\" (UID: \"3c89df4b-9274-4375-8ff8-47763bbf9182\") "
Oct 30 19:45:29.930842 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:29.930824    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c89df4b-9274-4375-8ff8-47763bbf9182-credential-socket" pod "3c89df4b-9274-4375-8ff8-47763bbf9182" (UID: "3c89df4b-9274-4375-8ff8-47763bbf9182"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:29.932922 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:29.931004    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c89df4b-9274-4375-8ff8-47763bbf9182-istiod-ca-cert" pod "3c89df4b-9274-4375-8ff8-47763bbf9182" (UID: "3c89df4b-9274-4375-8ff8-47763bbf9182"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:29.932922 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:29.931008    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c89df4b-9274-4375-8ff8-47763bbf9182-istio-data" pod "3c89df4b-9274-4375-8ff8-47763bbf9182" (UID: "3c89df4b-9274-4375-8ff8-47763bbf9182"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:29.932922 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:29.931068    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c89df4b-9274-4375-8ff8-47763bbf9182-workload-socket" pod "3c89df4b-9274-4375-8ff8-47763bbf9182" (UID: "3c89df4b-9274-4375-8ff8-47763bbf9182"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:29.932922 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:29.931279    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c89df4b-9274-4375-8ff8-47763bbf9182-workload-certs" pod "3c89df4b-9274-4375-8ff8-47763bbf9182" (UID: "3c89df4b-9274-4375-8ff8-47763bbf9182"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:29.932922 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:29.931306    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c89df4b-9274-4375-8ff8-47763bbf9182-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:29.932922 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:29.931337    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c89df4b-9274-4375-8ff8-47763bbf9182-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:29.932922 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:29.931352    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c89df4b-9274-4375-8ff8-47763bbf9182-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:29.932922 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:29.931367    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c89df4b-9274-4375-8ff8-47763bbf9182-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:29.933794 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:29.933636    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c89df4b-9274-4375-8ff8-47763bbf9182-istio-token" pod "3c89df4b-9274-4375-8ff8-47763bbf9182" (UID: "3c89df4b-9274-4375-8ff8-47763bbf9182"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:29.934228 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:29.934201    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c89df4b-9274-4375-8ff8-47763bbf9182-kube-api-access-kk2sb" pod "3c89df4b-9274-4375-8ff8-47763bbf9182" (UID: "3c89df4b-9274-4375-8ff8-47763bbf9182"). InnerVolumeSpecName "kube-api-access-kk2sb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:29.934688 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:29.934662    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c89df4b-9274-4375-8ff8-47763bbf9182-istio-envoy" pod "3c89df4b-9274-4375-8ff8-47763bbf9182" (UID: "3c89df4b-9274-4375-8ff8-47763bbf9182"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:29.935036 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:29.935016    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c89df4b-9274-4375-8ff8-47763bbf9182-istio-podinfo" pod "3c89df4b-9274-4375-8ff8-47763bbf9182" (UID: "3c89df4b-9274-4375-8ff8-47763bbf9182"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:45:30.034213 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:30.034144    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c89df4b-9274-4375-8ff8-47763bbf9182-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:30.034213 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:30.034172    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c89df4b-9274-4375-8ff8-47763bbf9182-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:30.034213 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:30.034182    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c89df4b-9274-4375-8ff8-47763bbf9182-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:30.034213 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:30.034190    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c89df4b-9274-4375-8ff8-47763bbf9182-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:30.034213 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:30.034200    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk2sb\" (UniqueName: \"kubernetes.io/projected/3c89df4b-9274-4375-8ff8-47763bbf9182-kube-api-access-kk2sb\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:30.559290 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:30.559249    5043 generic.go:356] "Generic (PLEG): container finished" podID="3c89df4b-9274-4375-8ff8-47763bbf9182" containerID="713483e9da389aa486b8e73e69d2718461ac01515c9c1c6aa61da6873afbc446" exitCode=0
Oct 30 19:45:30.559765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:30.559333    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq"
Oct 30 19:45:30.559765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:30.559381    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq" event={"ID":"3c89df4b-9274-4375-8ff8-47763bbf9182","Type":"ContainerDied","Data":"713483e9da389aa486b8e73e69d2718461ac01515c9c1c6aa61da6873afbc446"}
Oct 30 19:45:30.559765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:30.559443    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq" event={"ID":"3c89df4b-9274-4375-8ff8-47763bbf9182","Type":"ContainerDied","Data":"e5659a68adc7c15c5fa236a2346c671d132a82be1036ef7252e34afb0abe7feb"}
Oct 30 19:45:30.559765 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:30.559474    5043 scope.go:122] "RemoveContainer" containerID="713483e9da389aa486b8e73e69d2718461ac01515c9c1c6aa61da6873afbc446"
Oct 30 19:45:30.572962 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:30.572945    5043 scope.go:122] "RemoveContainer" containerID="713483e9da389aa486b8e73e69d2718461ac01515c9c1c6aa61da6873afbc446"
Oct 30 19:45:30.573276 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:45:30.573249    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"713483e9da389aa486b8e73e69d2718461ac01515c9c1c6aa61da6873afbc446\": container with ID starting with 713483e9da389aa486b8e73e69d2718461ac01515c9c1c6aa61da6873afbc446 not found: ID does not exist" containerID="713483e9da389aa486b8e73e69d2718461ac01515c9c1c6aa61da6873afbc446"
Oct 30 19:45:30.573388 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:30.573278    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"713483e9da389aa486b8e73e69d2718461ac01515c9c1c6aa61da6873afbc446"} err="failed to get container status \"713483e9da389aa486b8e73e69d2718461ac01515c9c1c6aa61da6873afbc446\": rpc error: code = NotFound desc = could not find container \"713483e9da389aa486b8e73e69d2718461ac01515c9c1c6aa61da6873afbc446\": container with ID starting with 713483e9da389aa486b8e73e69d2718461ac01515c9c1c6aa61da6873afbc446 not found: ID does not exist"
Oct 30 19:45:30.590539 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:30.590510    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq"]
Oct 30 19:45:30.598704 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:30.598676    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-jmxjq"]
Oct 30 19:45:30.680470 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:30.680430    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c89df4b-9274-4375-8ff8-47763bbf9182" path="/var/lib/kubelet/pods/3c89df4b-9274-4375-8ff8-47763bbf9182/volumes"
Oct 30 19:45:31.226916 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:31.226878    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3e35df-7874fb7595-2v6xk" podUID="a47d8311-eef7-47b2-b426-140198d456ee" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:45:34.529153 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:34.529124    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3e35df-7874fb7595-2v6xk"
Oct 30 19:45:34.589520 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:34.589486    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a47d8311-eef7-47b2-b426-140198d456ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a47d8311-eef7-47b2-b426-140198d456ee-plugins-config-volume\") pod \"a47d8311-eef7-47b2-b426-140198d456ee\" (UID: \"a47d8311-eef7-47b2-b426-140198d456ee\") "
Oct 30 19:45:34.589711 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:34.589549    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a47d8311-eef7-47b2-b426-140198d456ee-kube-api-access-jtzw5\" (UniqueName: \"kubernetes.io/projected/a47d8311-eef7-47b2-b426-140198d456ee-kube-api-access-jtzw5\") pod \"a47d8311-eef7-47b2-b426-140198d456ee\" (UID: \"a47d8311-eef7-47b2-b426-140198d456ee\") "
Oct 30 19:45:34.589935 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:34.589912    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a47d8311-eef7-47b2-b426-140198d456ee-plugins-config-volume" pod "a47d8311-eef7-47b2-b426-140198d456ee" (UID: "a47d8311-eef7-47b2-b426-140198d456ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:34.590051 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:34.590034    5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a47d8311-eef7-47b2-b426-140198d456ee-plugins-config-volume\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:34.592385 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:34.592353    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a47d8311-eef7-47b2-b426-140198d456ee-kube-api-access-jtzw5" pod "a47d8311-eef7-47b2-b426-140198d456ee" (UID: "a47d8311-eef7-47b2-b426-140198d456ee"). InnerVolumeSpecName "kube-api-access-jtzw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:34.596127 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:34.596102    5043 generic.go:356] "Generic (PLEG): container finished" podID="a47d8311-eef7-47b2-b426-140198d456ee" containerID="55451d2e63dcffbe130ecce3215310f15e8f4545b6160d879fc1f0b99386b22e" exitCode=0
Oct 30 19:45:34.596227 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:34.596151    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3e35df-7874fb7595-2v6xk"
Oct 30 19:45:34.596227 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:34.596166    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3e35df-7874fb7595-2v6xk" event={"ID":"a47d8311-eef7-47b2-b426-140198d456ee","Type":"ContainerDied","Data":"55451d2e63dcffbe130ecce3215310f15e8f4545b6160d879fc1f0b99386b22e"}
Oct 30 19:45:34.596227 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:34.596217    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3e35df-7874fb7595-2v6xk" event={"ID":"a47d8311-eef7-47b2-b426-140198d456ee","Type":"ContainerDied","Data":"28faac7eb52b795139e82858cfd36e6551360655c2cc26b6d6ca12e739ed106e"}
Oct 30 19:45:34.596331 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:34.596246    5043 scope.go:122] "RemoveContainer" containerID="55451d2e63dcffbe130ecce3215310f15e8f4545b6160d879fc1f0b99386b22e"
Oct 30 19:45:34.613416 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:34.613398    5043 scope.go:122] "RemoveContainer" containerID="55451d2e63dcffbe130ecce3215310f15e8f4545b6160d879fc1f0b99386b22e"
Oct 30 19:45:34.613696 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:45:34.613676    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55451d2e63dcffbe130ecce3215310f15e8f4545b6160d879fc1f0b99386b22e\": container with ID starting with 55451d2e63dcffbe130ecce3215310f15e8f4545b6160d879fc1f0b99386b22e not found: ID does not exist" containerID="55451d2e63dcffbe130ecce3215310f15e8f4545b6160d879fc1f0b99386b22e"
Oct 30 19:45:34.613788 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:34.613703    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55451d2e63dcffbe130ecce3215310f15e8f4545b6160d879fc1f0b99386b22e"} err="failed to get container status \"55451d2e63dcffbe130ecce3215310f15e8f4545b6160d879fc1f0b99386b22e\": rpc error: code = NotFound desc = could not find container \"55451d2e63dcffbe130ecce3215310f15e8f4545b6160d879fc1f0b99386b22e\": container with ID starting with 55451d2e63dcffbe130ecce3215310f15e8f4545b6160d879fc1f0b99386b22e not found: ID does not exist"
Oct 30 19:45:34.628314 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:34.628294    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3e35df-7874fb7595-2v6xk"]
Oct 30 19:45:34.632594 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:34.632566    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3e35df-7874fb7595-2v6xk"]
Oct 30 19:45:34.646220 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:34.646197    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a47d8311-eef7-47b2-b426-140198d456ee" path="/var/lib/kubelet/pods/a47d8311-eef7-47b2-b426-140198d456ee/volumes"
Oct 30 19:45:34.691529 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:34.691505    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtzw5\" (UniqueName: \"kubernetes.io/projected/a47d8311-eef7-47b2-b426-140198d456ee-kube-api-access-jtzw5\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:35.294530 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:35.294499    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f"]
Oct 30 19:45:35.294790 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:35.294765    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f" podUID="abc3d11d-936f-41d9-a35f-adf6ab7f7909" containerName="istio-proxy" containerID="cri-o://5df6a3c856c9ec8f23289573e39bb1dbae89bebdf070aa296599acc7e1a65525" gracePeriod=30
Oct 30 19:45:35.301833 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:35.301795    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc"]
Oct 30 19:45:35.302000 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:35.301983    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc" podUID="7af56765-aede-4480-8b11-83646e0f5284" containerName="istio-proxy" containerID="cri-o://ccd2f4229ca1ee1682bef8db4f483e81232ed682ae0fa15d1f6b8c43ea141508" gracePeriod=30
Oct 30 19:45:35.424696 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:35.424653    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-47110f-65f4f5b6b4-5528c"]
Oct 30 19:45:35.443333 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:35.443287    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-47110f-7cdc4b47f5-2brw4"]
Oct 30 19:45:35.450177 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:35.449526    5043 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:45:35.450177 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:35.449665    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc" podUID="7af56765-aede-4480-8b11-83646e0f5284" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:45:37.531288 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:37.531240    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-47110f-mc9ph"]
Oct 30 19:45:37.534443 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:37.534416    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-47110f-mc9ph"]
Oct 30 19:45:38.378195 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:38.378161    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr"]
Oct 30 19:45:38.378432 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:38.378407    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr" podUID="5881801f-8f93-4d5f-b303-2c89af131c12" containerName="istio-proxy" containerID="cri-o://6356573d12eb697722f991ee6d9aed500fafd31e7e2f49bc3f8521d6b5be8127" gracePeriod=30
Oct 30 19:45:38.646098 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:38.646005    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6a8f7ef-e408-43d3-8816-9483b8754819" path="/var/lib/kubelet/pods/f6a8f7ef-e408-43d3-8816-9483b8754819/volumes"
Oct 30 19:45:39.084910 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:39.084871    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ce69f3-f6rwg"]
Oct 30 19:45:39.085395 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:39.085379    5043 cpu_manager.go:414] "containerMap: removing container" podUID="3c89df4b-9274-4375-8ff8-47763bbf9182" containerName="istio-proxy"
Oct 30 19:45:39.085395 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:39.085395    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c89df4b-9274-4375-8ff8-47763bbf9182" containerName="istio-proxy"
Oct 30 19:45:39.085572 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:39.085410    5043 cpu_manager.go:414] "containerMap: removing container" podUID="741a0acd-9951-4c23-abf6-1c9c22e15366" containerName="payload-processing"
Oct 30 19:45:39.085572 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:39.085415    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="741a0acd-9951-4c23-abf6-1c9c22e15366" containerName="payload-processing"
Oct 30 19:45:39.085572 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:39.085428    5043 cpu_manager.go:414] "containerMap: removing container" podUID="6012f92c-1834-4e04-8d81-5a8ea39d394c" containerName="cleanup"
Oct 30 19:45:39.085572 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:39.085433    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6012f92c-1834-4e04-8d81-5a8ea39d394c" containerName="cleanup"
Oct 30 19:45:39.085572 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:39.085448    5043 cpu_manager.go:414] "containerMap: removing container" podUID="834c26fa-e19d-4db2-92ea-331d49a139aa" containerName="cleanup"
Oct 30 19:45:39.085572 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:39.085452    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="834c26fa-e19d-4db2-92ea-331d49a139aa" containerName="cleanup"
Oct 30 19:45:39.085572 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:39.085468    5043 cpu_manager.go:414] "containerMap: removing container" podUID="cedabb17-f317-491f-ae1b-0b5b51a80cd6" containerName="cleanup"
Oct 30 19:45:39.085572 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:39.085474    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cedabb17-f317-491f-ae1b-0b5b51a80cd6" containerName="cleanup"
Oct 30 19:45:39.085572 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:39.085484    5043 cpu_manager.go:414] "containerMap: removing container" podUID="f6a8f7ef-e408-43d3-8816-9483b8754819" containerName="revoke-keys"
Oct 30 19:45:39.085572 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:39.085490    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6a8f7ef-e408-43d3-8816-9483b8754819" containerName="revoke-keys"
Oct 30 19:45:39.085572 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:39.085519    5043 cpu_manager.go:414] "containerMap: removing container" podUID="a47d8311-eef7-47b2-b426-140198d456ee" containerName="payload-pre-processing"
Oct 30 19:45:39.085572 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:39.085524    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a47d8311-eef7-47b2-b426-140198d456ee" containerName="payload-pre-processing"
Oct 30 19:45:39.086533 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:39.085621    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6a8f7ef-e408-43d3-8816-9483b8754819" containerName="revoke-keys"
Oct 30 19:45:39.086533 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:39.085633    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="a47d8311-eef7-47b2-b426-140198d456ee" containerName="payload-pre-processing"
Oct 30 19:45:39.086533 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:39.085640    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c89df4b-9274-4375-8ff8-47763bbf9182" containerName="istio-proxy"
Oct 30 19:45:39.086533 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:39.085648    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf5e794c-4d01-4ab8-b211-1d2346d9b82b" containerName="cleanup"
Oct 30 19:45:39.086533 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:39.085653    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="741a0acd-9951-4c23-abf6-1c9c22e15366" containerName="payload-processing"
Oct 30 19:45:39.086533 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:39.085658    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="834c26fa-e19d-4db2-92ea-331d49a139aa" containerName="cleanup"
Oct 30 19:45:39.086533 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:39.085663    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="6012f92c-1834-4e04-8d81-5a8ea39d394c" containerName="cleanup"
Oct 30 19:45:39.089146 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:39.089128    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ce69f3-f6rwg"
Oct 30 19:45:39.101685 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:39.101658    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:45:39.101874 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:39.101728    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:45:39.102324 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:39.102306    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:45:39.102384 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:39.102371    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:45:39.103422 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:39.103394    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ce69f3-f6rwg"]
Oct 30 19:45:39.153523 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:39.153482    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlgr7\" (UniqueName: \"kubernetes.io/projected/30f3ac96-552e-40b3-8f49-eea750ba8584-kube-api-access-rlgr7\") pod \"maas-api-revoke-keys-e2e-worker-w3-ce69f3-f6rwg\" (UID: \"30f3ac96-552e-40b3-8f49-eea750ba8584\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ce69f3-f6rwg"
Oct 30 19:45:39.153702 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:39.153534    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30f3ac96-552e-40b3-8f49-eea750ba8584-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ce69f3-f6rwg\" (UID: \"30f3ac96-552e-40b3-8f49-eea750ba8584\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ce69f3-f6rwg"
Oct 30 19:45:39.153758 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:39.153728    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30f3ac96-552e-40b3-8f49-eea750ba8584-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ce69f3-f6rwg\" (UID: \"30f3ac96-552e-40b3-8f49-eea750ba8584\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ce69f3-f6rwg"
Oct 30 19:45:39.255269 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:39.255232    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlgr7\" (UniqueName: \"kubernetes.io/projected/30f3ac96-552e-40b3-8f49-eea750ba8584-kube-api-access-rlgr7\") pod \"maas-api-revoke-keys-e2e-worker-w3-ce69f3-f6rwg\" (UID: \"30f3ac96-552e-40b3-8f49-eea750ba8584\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ce69f3-f6rwg"
Oct 30 19:45:39.255269 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:39.255275    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30f3ac96-552e-40b3-8f49-eea750ba8584-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ce69f3-f6rwg\" (UID: \"30f3ac96-552e-40b3-8f49-eea750ba8584\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ce69f3-f6rwg"
Oct 30 19:45:39.255532 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:39.255396    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30f3ac96-552e-40b3-8f49-eea750ba8584-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ce69f3-f6rwg\" (UID: \"30f3ac96-552e-40b3-8f49-eea750ba8584\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ce69f3-f6rwg"
Oct 30 19:45:39.256065 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:39.256039    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30f3ac96-552e-40b3-8f49-eea750ba8584-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ce69f3-f6rwg\" (UID: \"30f3ac96-552e-40b3-8f49-eea750ba8584\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ce69f3-f6rwg"
Oct 30 19:45:39.258064 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:39.258036    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30f3ac96-552e-40b3-8f49-eea750ba8584-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ce69f3-f6rwg\" (UID: \"30f3ac96-552e-40b3-8f49-eea750ba8584\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ce69f3-f6rwg"
Oct 30 19:45:39.267788 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:39.267730    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlgr7\" (UniqueName: \"kubernetes.io/projected/30f3ac96-552e-40b3-8f49-eea750ba8584-kube-api-access-rlgr7\") pod \"maas-api-revoke-keys-e2e-worker-w3-ce69f3-f6rwg\" (UID: \"30f3ac96-552e-40b3-8f49-eea750ba8584\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ce69f3-f6rwg"
Oct 30 19:45:39.405948 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:39.405866    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ce69f3-f6rwg"
Oct 30 19:45:39.703086 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:39.701126    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ce69f3-f6rwg"]
Oct 30 19:45:40.425483 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.425425    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-47110f-65f4f5b6b4-5528c" podUID="ceac8507-3941-431a-9880-2fb98e79e257" containerName="payload-processing" containerID="cri-o://235892c7afed574dbfdf735db599ba73f5cace485be26b1172bd1bc281fa23e0" gracePeriod=25
Oct 30 19:45:40.444722 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.444038    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-47110f-7cdc4b47f5-2brw4" podUID="83c175fe-cdfd-4a54-88b6-f9a2061a11dd" containerName="payload-pre-processing" containerID="cri-o://6f7549fab0ce5ce26f5fd1a06edaaacc6b005037e60cac8a11744b52ceea452f" gracePeriod=25
Oct 30 19:45:40.584911 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.584807    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f"
Oct 30 19:45:40.653585 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.653552    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc"
Oct 30 19:45:40.692762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.690289    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abc3d11d-936f-41d9-a35f-adf6ab7f7909-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc3d11d-936f-41d9-a35f-adf6ab7f7909-workload-socket\") pod \"abc3d11d-936f-41d9-a35f-adf6ab7f7909\" (UID: \"abc3d11d-936f-41d9-a35f-adf6ab7f7909\") "
Oct 30 19:45:40.692762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.690342    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abc3d11d-936f-41d9-a35f-adf6ab7f7909-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc3d11d-936f-41d9-a35f-adf6ab7f7909-credential-socket\") pod \"abc3d11d-936f-41d9-a35f-adf6ab7f7909\" (UID: \"abc3d11d-936f-41d9-a35f-adf6ab7f7909\") "
Oct 30 19:45:40.692762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.690498    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abc3d11d-936f-41d9-a35f-adf6ab7f7909-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abc3d11d-936f-41d9-a35f-adf6ab7f7909-istio-data\") pod \"abc3d11d-936f-41d9-a35f-adf6ab7f7909\" (UID: \"abc3d11d-936f-41d9-a35f-adf6ab7f7909\") "
Oct 30 19:45:40.692762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.690535    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abc3d11d-936f-41d9-a35f-adf6ab7f7909-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abc3d11d-936f-41d9-a35f-adf6ab7f7909-workload-certs\") pod \"abc3d11d-936f-41d9-a35f-adf6ab7f7909\" (UID: \"abc3d11d-936f-41d9-a35f-adf6ab7f7909\") "
Oct 30 19:45:40.692762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.690569    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abc3d11d-936f-41d9-a35f-adf6ab7f7909-kube-api-access-kcs6p\" (UniqueName: \"kubernetes.io/projected/abc3d11d-936f-41d9-a35f-adf6ab7f7909-kube-api-access-kcs6p\") pod \"abc3d11d-936f-41d9-a35f-adf6ab7f7909\" (UID: \"abc3d11d-936f-41d9-a35f-adf6ab7f7909\") "
Oct 30 19:45:40.692762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.690617    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abc3d11d-936f-41d9-a35f-adf6ab7f7909-istio-token\" (UniqueName: \"kubernetes.io/projected/abc3d11d-936f-41d9-a35f-adf6ab7f7909-istio-token\") pod \"abc3d11d-936f-41d9-a35f-adf6ab7f7909\" (UID: \"abc3d11d-936f-41d9-a35f-adf6ab7f7909\") "
Oct 30 19:45:40.692762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.690701    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abc3d11d-936f-41d9-a35f-adf6ab7f7909-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abc3d11d-936f-41d9-a35f-adf6ab7f7909-istiod-ca-cert\") pod \"abc3d11d-936f-41d9-a35f-adf6ab7f7909\" (UID: \"abc3d11d-936f-41d9-a35f-adf6ab7f7909\") "
Oct 30 19:45:40.692762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.690761    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/abc3d11d-936f-41d9-a35f-adf6ab7f7909-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abc3d11d-936f-41d9-a35f-adf6ab7f7909-istio-podinfo\") pod \"abc3d11d-936f-41d9-a35f-adf6ab7f7909\" (UID: \"abc3d11d-936f-41d9-a35f-adf6ab7f7909\") "
Oct 30 19:45:40.692762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.690791    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abc3d11d-936f-41d9-a35f-adf6ab7f7909-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abc3d11d-936f-41d9-a35f-adf6ab7f7909-istio-envoy\") pod \"abc3d11d-936f-41d9-a35f-adf6ab7f7909\" (UID: \"abc3d11d-936f-41d9-a35f-adf6ab7f7909\") "
Oct 30 19:45:40.692762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.691884    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc3d11d-936f-41d9-a35f-adf6ab7f7909-workload-certs" pod "abc3d11d-936f-41d9-a35f-adf6ab7f7909" (UID: "abc3d11d-936f-41d9-a35f-adf6ab7f7909"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:40.692762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.692546    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc3d11d-936f-41d9-a35f-adf6ab7f7909-workload-socket" pod "abc3d11d-936f-41d9-a35f-adf6ab7f7909" (UID: "abc3d11d-936f-41d9-a35f-adf6ab7f7909"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:40.692762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.692646    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abc3d11d-936f-41d9-a35f-adf6ab7f7909-istiod-ca-cert" pod "abc3d11d-936f-41d9-a35f-adf6ab7f7909" (UID: "abc3d11d-936f-41d9-a35f-adf6ab7f7909"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:40.693566 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.693013    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc3d11d-936f-41d9-a35f-adf6ab7f7909-istio-data" pod "abc3d11d-936f-41d9-a35f-adf6ab7f7909" (UID: "abc3d11d-936f-41d9-a35f-adf6ab7f7909"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:40.693566 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.693248    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc3d11d-936f-41d9-a35f-adf6ab7f7909-credential-socket" pod "abc3d11d-936f-41d9-a35f-adf6ab7f7909" (UID: "abc3d11d-936f-41d9-a35f-adf6ab7f7909"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:40.697788 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.694455    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc3d11d-936f-41d9-a35f-adf6ab7f7909-istio-token" pod "abc3d11d-936f-41d9-a35f-adf6ab7f7909" (UID: "abc3d11d-936f-41d9-a35f-adf6ab7f7909"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:40.697788 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.695129    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abc3d11d-936f-41d9-a35f-adf6ab7f7909-istio-podinfo" pod "abc3d11d-936f-41d9-a35f-adf6ab7f7909" (UID: "abc3d11d-936f-41d9-a35f-adf6ab7f7909"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:45:40.697788 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.695809    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc3d11d-936f-41d9-a35f-adf6ab7f7909-kube-api-access-kcs6p" pod "abc3d11d-936f-41d9-a35f-adf6ab7f7909" (UID: "abc3d11d-936f-41d9-a35f-adf6ab7f7909"). InnerVolumeSpecName "kube-api-access-kcs6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:40.698050 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.697946    5043 generic.go:356] "Generic (PLEG): container finished" podID="ceac8507-3941-431a-9880-2fb98e79e257" containerID="235892c7afed574dbfdf735db599ba73f5cace485be26b1172bd1bc281fa23e0" exitCode=0
Oct 30 19:45:40.698506 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.698177    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-47110f-65f4f5b6b4-5528c" event={"ID":"ceac8507-3941-431a-9880-2fb98e79e257","Type":"ContainerDied","Data":"235892c7afed574dbfdf735db599ba73f5cace485be26b1172bd1bc281fa23e0"}
Oct 30 19:45:40.700881 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.700170    5043 generic.go:356] "Generic (PLEG): container finished" podID="30f3ac96-552e-40b3-8f49-eea750ba8584" containerID="e36957a61cbe6d280391d5deb6d1289bf83c419080e33092f13b4f303112114c" exitCode=0
Oct 30 19:45:40.700881 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.700238    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ce69f3-f6rwg" event={"ID":"30f3ac96-552e-40b3-8f49-eea750ba8584","Type":"ContainerDied","Data":"e36957a61cbe6d280391d5deb6d1289bf83c419080e33092f13b4f303112114c"}
Oct 30 19:45:40.700881 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.700269    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ce69f3-f6rwg" event={"ID":"30f3ac96-552e-40b3-8f49-eea750ba8584","Type":"ContainerStarted","Data":"1ddd528f98fc8534d5bbe04248254495d50e01af0e3f021c4501ea8c5e3e6384"}
Oct 30 19:45:40.701913 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.701889    5043 generic.go:356] "Generic (PLEG): container finished" podID="7af56765-aede-4480-8b11-83646e0f5284" containerID="ccd2f4229ca1ee1682bef8db4f483e81232ed682ae0fa15d1f6b8c43ea141508" exitCode=0
Oct 30 19:45:40.702269 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.702059    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc"
Oct 30 19:45:40.702269 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.702092    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc" event={"ID":"7af56765-aede-4480-8b11-83646e0f5284","Type":"ContainerDied","Data":"ccd2f4229ca1ee1682bef8db4f483e81232ed682ae0fa15d1f6b8c43ea141508"}
Oct 30 19:45:40.702269 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.702137    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc" event={"ID":"7af56765-aede-4480-8b11-83646e0f5284","Type":"ContainerDied","Data":"e6350aa154576a6431d096e157c1e237c1c0b6053eef9b1e57a280006196f37f"}
Oct 30 19:45:40.702269 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.702168    5043 scope.go:122] "RemoveContainer" containerID="ccd2f4229ca1ee1682bef8db4f483e81232ed682ae0fa15d1f6b8c43ea141508"
Oct 30 19:45:40.704599 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.704579    5043 generic.go:356] "Generic (PLEG): container finished" podID="abc3d11d-936f-41d9-a35f-adf6ab7f7909" containerID="5df6a3c856c9ec8f23289573e39bb1dbae89bebdf070aa296599acc7e1a65525" exitCode=0
Oct 30 19:45:40.704993 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.704641    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f" event={"ID":"abc3d11d-936f-41d9-a35f-adf6ab7f7909","Type":"ContainerDied","Data":"5df6a3c856c9ec8f23289573e39bb1dbae89bebdf070aa296599acc7e1a65525"}
Oct 30 19:45:40.704993 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.704764    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f" event={"ID":"abc3d11d-936f-41d9-a35f-adf6ab7f7909","Type":"ContainerDied","Data":"0a759aaae016799c70b64f9d4209875160c221275ae45d16e760738cddc84b6b"}
Oct 30 19:45:40.704993 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.704846    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f"
Oct 30 19:45:40.710142 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.710114    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc3d11d-936f-41d9-a35f-adf6ab7f7909-istio-envoy" pod "abc3d11d-936f-41d9-a35f-adf6ab7f7909" (UID: "abc3d11d-936f-41d9-a35f-adf6ab7f7909"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:40.713004 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.712979    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-47110f-65f4f5b6b4-5528c"
Oct 30 19:45:40.716237 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.716220    5043 scope.go:122] "RemoveContainer" containerID="ccd2f4229ca1ee1682bef8db4f483e81232ed682ae0fa15d1f6b8c43ea141508"
Oct 30 19:45:40.716505 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:45:40.716486    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccd2f4229ca1ee1682bef8db4f483e81232ed682ae0fa15d1f6b8c43ea141508\": container with ID starting with ccd2f4229ca1ee1682bef8db4f483e81232ed682ae0fa15d1f6b8c43ea141508 not found: ID does not exist" containerID="ccd2f4229ca1ee1682bef8db4f483e81232ed682ae0fa15d1f6b8c43ea141508"
Oct 30 19:45:40.716571 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.716513    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccd2f4229ca1ee1682bef8db4f483e81232ed682ae0fa15d1f6b8c43ea141508"} err="failed to get container status \"ccd2f4229ca1ee1682bef8db4f483e81232ed682ae0fa15d1f6b8c43ea141508\": rpc error: code = NotFound desc = could not find container \"ccd2f4229ca1ee1682bef8db4f483e81232ed682ae0fa15d1f6b8c43ea141508\": container with ID starting with ccd2f4229ca1ee1682bef8db4f483e81232ed682ae0fa15d1f6b8c43ea141508 not found: ID does not exist"
Oct 30 19:45:40.716571 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.716529    5043 scope.go:122] "RemoveContainer" containerID="5df6a3c856c9ec8f23289573e39bb1dbae89bebdf070aa296599acc7e1a65525"
Oct 30 19:45:40.726215 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.726197    5043 scope.go:122] "RemoveContainer" containerID="ec24646caaf543e078ff439bd52db9f9346d5fef8bca7acb0ddfc9faa3f2908e"
Oct 30 19:45:40.735974 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.735955    5043 scope.go:122] "RemoveContainer" containerID="5df6a3c856c9ec8f23289573e39bb1dbae89bebdf070aa296599acc7e1a65525"
Oct 30 19:45:40.736233 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:45:40.736215    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5df6a3c856c9ec8f23289573e39bb1dbae89bebdf070aa296599acc7e1a65525\": container with ID starting with 5df6a3c856c9ec8f23289573e39bb1dbae89bebdf070aa296599acc7e1a65525 not found: ID does not exist" containerID="5df6a3c856c9ec8f23289573e39bb1dbae89bebdf070aa296599acc7e1a65525"
Oct 30 19:45:40.736309 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.736246    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5df6a3c856c9ec8f23289573e39bb1dbae89bebdf070aa296599acc7e1a65525"} err="failed to get container status \"5df6a3c856c9ec8f23289573e39bb1dbae89bebdf070aa296599acc7e1a65525\": rpc error: code = NotFound desc = could not find container \"5df6a3c856c9ec8f23289573e39bb1dbae89bebdf070aa296599acc7e1a65525\": container with ID starting with 5df6a3c856c9ec8f23289573e39bb1dbae89bebdf070aa296599acc7e1a65525 not found: ID does not exist"
Oct 30 19:45:40.736309 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.736268    5043 scope.go:122] "RemoveContainer" containerID="ec24646caaf543e078ff439bd52db9f9346d5fef8bca7acb0ddfc9faa3f2908e"
Oct 30 19:45:40.736495 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:45:40.736478    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec24646caaf543e078ff439bd52db9f9346d5fef8bca7acb0ddfc9faa3f2908e\": container with ID starting with ec24646caaf543e078ff439bd52db9f9346d5fef8bca7acb0ddfc9faa3f2908e not found: ID does not exist" containerID="ec24646caaf543e078ff439bd52db9f9346d5fef8bca7acb0ddfc9faa3f2908e"
Oct 30 19:45:40.736550 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.736501    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec24646caaf543e078ff439bd52db9f9346d5fef8bca7acb0ddfc9faa3f2908e"} err="failed to get container status \"ec24646caaf543e078ff439bd52db9f9346d5fef8bca7acb0ddfc9faa3f2908e\": rpc error: code = NotFound desc = could not find container \"ec24646caaf543e078ff439bd52db9f9346d5fef8bca7acb0ddfc9faa3f2908e\": container with ID starting with ec24646caaf543e078ff439bd52db9f9346d5fef8bca7acb0ddfc9faa3f2908e not found: ID does not exist"
Oct 30 19:45:40.791693 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.791659    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7af56765-aede-4480-8b11-83646e0f5284-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af56765-aede-4480-8b11-83646e0f5284-credential-socket\") pod \"7af56765-aede-4480-8b11-83646e0f5284\" (UID: \"7af56765-aede-4480-8b11-83646e0f5284\") "
Oct 30 19:45:40.791882 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.791722    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7af56765-aede-4480-8b11-83646e0f5284-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7af56765-aede-4480-8b11-83646e0f5284-istio-data\") pod \"7af56765-aede-4480-8b11-83646e0f5284\" (UID: \"7af56765-aede-4480-8b11-83646e0f5284\") "
Oct 30 19:45:40.791882 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.791792    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7af56765-aede-4480-8b11-83646e0f5284-istio-token\" (UniqueName: \"kubernetes.io/projected/7af56765-aede-4480-8b11-83646e0f5284-istio-token\") pod \"7af56765-aede-4480-8b11-83646e0f5284\" (UID: \"7af56765-aede-4480-8b11-83646e0f5284\") "
Oct 30 19:45:40.791882 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.791865    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ceac8507-3941-431a-9880-2fb98e79e257-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceac8507-3941-431a-9880-2fb98e79e257-plugins-config-volume\") pod \"ceac8507-3941-431a-9880-2fb98e79e257\" (UID: \"ceac8507-3941-431a-9880-2fb98e79e257\") "
Oct 30 19:45:40.792074 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.791925    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7af56765-aede-4480-8b11-83646e0f5284-kube-api-access-hfjlk\" (UniqueName: \"kubernetes.io/projected/7af56765-aede-4480-8b11-83646e0f5284-kube-api-access-hfjlk\") pod \"7af56765-aede-4480-8b11-83646e0f5284\" (UID: \"7af56765-aede-4480-8b11-83646e0f5284\") "
Oct 30 19:45:40.792074 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.792022    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7af56765-aede-4480-8b11-83646e0f5284-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7af56765-aede-4480-8b11-83646e0f5284-istio-podinfo\") pod \"7af56765-aede-4480-8b11-83646e0f5284\" (UID: \"7af56765-aede-4480-8b11-83646e0f5284\") "
Oct 30 19:45:40.792074 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.792063    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7af56765-aede-4480-8b11-83646e0f5284-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7af56765-aede-4480-8b11-83646e0f5284-istio-envoy\") pod \"7af56765-aede-4480-8b11-83646e0f5284\" (UID: \"7af56765-aede-4480-8b11-83646e0f5284\") "
Oct 30 19:45:40.792231 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.792110    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7af56765-aede-4480-8b11-83646e0f5284-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7af56765-aede-4480-8b11-83646e0f5284-workload-certs\") pod \"7af56765-aede-4480-8b11-83646e0f5284\" (UID: \"7af56765-aede-4480-8b11-83646e0f5284\") "
Oct 30 19:45:40.792405 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.792246    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceac8507-3941-431a-9880-2fb98e79e257-kube-api-access-52tgp\" (UniqueName: \"kubernetes.io/projected/ceac8507-3941-431a-9880-2fb98e79e257-kube-api-access-52tgp\") pod \"ceac8507-3941-431a-9880-2fb98e79e257\" (UID: \"ceac8507-3941-431a-9880-2fb98e79e257\") "
Oct 30 19:45:40.792405 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.792297    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7af56765-aede-4480-8b11-83646e0f5284-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af56765-aede-4480-8b11-83646e0f5284-workload-socket\") pod \"7af56765-aede-4480-8b11-83646e0f5284\" (UID: \"7af56765-aede-4480-8b11-83646e0f5284\") "
Oct 30 19:45:40.792405 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.792345    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7af56765-aede-4480-8b11-83646e0f5284-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7af56765-aede-4480-8b11-83646e0f5284-istiod-ca-cert\") pod \"7af56765-aede-4480-8b11-83646e0f5284\" (UID: \"7af56765-aede-4480-8b11-83646e0f5284\") "
Oct 30 19:45:40.794597 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.793980    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7af56765-aede-4480-8b11-83646e0f5284-workload-certs" pod "7af56765-aede-4480-8b11-83646e0f5284" (UID: "7af56765-aede-4480-8b11-83646e0f5284"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:40.794597 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.794031    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7af56765-aede-4480-8b11-83646e0f5284-workload-socket" pod "7af56765-aede-4480-8b11-83646e0f5284" (UID: "7af56765-aede-4480-8b11-83646e0f5284"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:40.794597 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.794184    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7af56765-aede-4480-8b11-83646e0f5284-istiod-ca-cert" pod "7af56765-aede-4480-8b11-83646e0f5284" (UID: "7af56765-aede-4480-8b11-83646e0f5284"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:40.794597 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.794272    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7af56765-aede-4480-8b11-83646e0f5284-istio-data" pod "7af56765-aede-4480-8b11-83646e0f5284" (UID: "7af56765-aede-4480-8b11-83646e0f5284"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:40.794597 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.794327    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7af56765-aede-4480-8b11-83646e0f5284-credential-socket" pod "7af56765-aede-4480-8b11-83646e0f5284" (UID: "7af56765-aede-4480-8b11-83646e0f5284"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:40.794977 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.794640    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceac8507-3941-431a-9880-2fb98e79e257-plugins-config-volume" pod "ceac8507-3941-431a-9880-2fb98e79e257" (UID: "ceac8507-3941-431a-9880-2fb98e79e257"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:40.795794 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.795552    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abc3d11d-936f-41d9-a35f-adf6ab7f7909-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:40.795794 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.795581    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcs6p\" (UniqueName: \"kubernetes.io/projected/abc3d11d-936f-41d9-a35f-adf6ab7f7909-kube-api-access-kcs6p\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:40.795794 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.795597    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abc3d11d-936f-41d9-a35f-adf6ab7f7909-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:40.795794 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.795610    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af56765-aede-4480-8b11-83646e0f5284-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:40.795794 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.795625    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7af56765-aede-4480-8b11-83646e0f5284-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:40.795794 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.795637    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abc3d11d-936f-41d9-a35f-adf6ab7f7909-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:40.795794 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.795650    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af56765-aede-4480-8b11-83646e0f5284-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:40.795794 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.795664    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abc3d11d-936f-41d9-a35f-adf6ab7f7909-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:40.795794 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.795677    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abc3d11d-936f-41d9-a35f-adf6ab7f7909-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:40.795794 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.795689    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7af56765-aede-4480-8b11-83646e0f5284-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:40.795794 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.795703    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc3d11d-936f-41d9-a35f-adf6ab7f7909-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:40.795794 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.795716    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc3d11d-936f-41d9-a35f-adf6ab7f7909-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:40.795794 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.795728    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7af56765-aede-4480-8b11-83646e0f5284-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:40.795794 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.795767    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abc3d11d-936f-41d9-a35f-adf6ab7f7909-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:40.798205 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.798173    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7af56765-aede-4480-8b11-83646e0f5284-istio-token" pod "7af56765-aede-4480-8b11-83646e0f5284" (UID: "7af56765-aede-4480-8b11-83646e0f5284"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:40.798439 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.798414    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7af56765-aede-4480-8b11-83646e0f5284-istio-podinfo" pod "7af56765-aede-4480-8b11-83646e0f5284" (UID: "7af56765-aede-4480-8b11-83646e0f5284"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:45:40.798651 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.798606    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7af56765-aede-4480-8b11-83646e0f5284-istio-envoy" pod "7af56765-aede-4480-8b11-83646e0f5284" (UID: "7af56765-aede-4480-8b11-83646e0f5284"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:40.798925 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.798900    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7af56765-aede-4480-8b11-83646e0f5284-kube-api-access-hfjlk" pod "7af56765-aede-4480-8b11-83646e0f5284" (UID: "7af56765-aede-4480-8b11-83646e0f5284"). InnerVolumeSpecName "kube-api-access-hfjlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:40.799333 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.799311    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceac8507-3941-431a-9880-2fb98e79e257-kube-api-access-52tgp" pod "ceac8507-3941-431a-9880-2fb98e79e257" (UID: "ceac8507-3941-431a-9880-2fb98e79e257"). InnerVolumeSpecName "kube-api-access-52tgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:40.897500 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.897406    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7af56765-aede-4480-8b11-83646e0f5284-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:40.897500 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.897442    5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceac8507-3941-431a-9880-2fb98e79e257-plugins-config-volume\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:40.897500 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.897460    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfjlk\" (UniqueName: \"kubernetes.io/projected/7af56765-aede-4480-8b11-83646e0f5284-kube-api-access-hfjlk\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:40.897500 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.897480    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7af56765-aede-4480-8b11-83646e0f5284-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:40.897500 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.897493    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7af56765-aede-4480-8b11-83646e0f5284-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:40.897790 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:40.897507    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52tgp\" (UniqueName: \"kubernetes.io/projected/ceac8507-3941-431a-9880-2fb98e79e257-kube-api-access-52tgp\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:41.054526 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:41.054494    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f"]
Oct 30 19:45:41.071286 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:41.071258    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-lr27f"]
Oct 30 19:45:41.099199 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:41.099164    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc"]
Oct 30 19:45:41.120713 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:41.120673    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-rkbbc"]
Oct 30 19:45:41.717353 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:41.717325    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-47110f-65f4f5b6b4-5528c"
Oct 30 19:45:41.717800 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:41.717370    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-47110f-65f4f5b6b4-5528c" event={"ID":"ceac8507-3941-431a-9880-2fb98e79e257","Type":"ContainerDied","Data":"200a1167da944ab0a4e2931a3e4fe1ecb1b29cc20f275529453eb8efd2a00d88"}
Oct 30 19:45:41.717800 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:41.717428    5043 scope.go:122] "RemoveContainer" containerID="235892c7afed574dbfdf735db599ba73f5cace485be26b1172bd1bc281fa23e0"
Oct 30 19:45:41.754443 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:41.754412    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-47110f-65f4f5b6b4-5528c"]
Oct 30 19:45:41.759344 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:41.759311    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-47110f-65f4f5b6b4-5528c"]
Oct 30 19:45:41.889596 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:41.889567    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ce69f3-f6rwg"
Oct 30 19:45:42.031068 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:42.030980    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30f3ac96-552e-40b3-8f49-eea750ba8584-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30f3ac96-552e-40b3-8f49-eea750ba8584-maas-api-xks-ca\") pod \"30f3ac96-552e-40b3-8f49-eea750ba8584\" (UID: \"30f3ac96-552e-40b3-8f49-eea750ba8584\") "
Oct 30 19:45:42.031068 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:42.031029    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30f3ac96-552e-40b3-8f49-eea750ba8584-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30f3ac96-552e-40b3-8f49-eea750ba8584-maas-api-openshift-ca\") pod \"30f3ac96-552e-40b3-8f49-eea750ba8584\" (UID: \"30f3ac96-552e-40b3-8f49-eea750ba8584\") "
Oct 30 19:45:42.031302 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:42.031100    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30f3ac96-552e-40b3-8f49-eea750ba8584-kube-api-access-rlgr7\" (UniqueName: \"kubernetes.io/projected/30f3ac96-552e-40b3-8f49-eea750ba8584-kube-api-access-rlgr7\") pod \"30f3ac96-552e-40b3-8f49-eea750ba8584\" (UID: \"30f3ac96-552e-40b3-8f49-eea750ba8584\") "
Oct 30 19:45:42.031522 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:42.031474    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30f3ac96-552e-40b3-8f49-eea750ba8584-maas-api-openshift-ca" pod "30f3ac96-552e-40b3-8f49-eea750ba8584" (UID: "30f3ac96-552e-40b3-8f49-eea750ba8584"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:42.032276 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:42.032223    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30f3ac96-552e-40b3-8f49-eea750ba8584-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:42.041283 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:42.041253    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30f3ac96-552e-40b3-8f49-eea750ba8584-kube-api-access-rlgr7" pod "30f3ac96-552e-40b3-8f49-eea750ba8584" (UID: "30f3ac96-552e-40b3-8f49-eea750ba8584"). InnerVolumeSpecName "kube-api-access-rlgr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:42.041711 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:42.041680    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30f3ac96-552e-40b3-8f49-eea750ba8584-maas-api-xks-ca" pod "30f3ac96-552e-40b3-8f49-eea750ba8584" (UID: "30f3ac96-552e-40b3-8f49-eea750ba8584"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:45:42.133570 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:42.133539    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlgr7\" (UniqueName: \"kubernetes.io/projected/30f3ac96-552e-40b3-8f49-eea750ba8584-kube-api-access-rlgr7\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:42.133570 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:42.133569    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30f3ac96-552e-40b3-8f49-eea750ba8584-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:42.646147 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:42.646115    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7af56765-aede-4480-8b11-83646e0f5284" path="/var/lib/kubelet/pods/7af56765-aede-4480-8b11-83646e0f5284/volumes"
Oct 30 19:45:42.646562 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:42.646547    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abc3d11d-936f-41d9-a35f-adf6ab7f7909" path="/var/lib/kubelet/pods/abc3d11d-936f-41d9-a35f-adf6ab7f7909/volumes"
Oct 30 19:45:42.646938 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:42.646922    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ceac8507-3941-431a-9880-2fb98e79e257" path="/var/lib/kubelet/pods/ceac8507-3941-431a-9880-2fb98e79e257/volumes"
Oct 30 19:45:42.726311 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:42.726285    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ce69f3-f6rwg"
Oct 30 19:45:42.726708 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:42.726321    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ce69f3-f6rwg" event={"ID":"30f3ac96-552e-40b3-8f49-eea750ba8584","Type":"ContainerDied","Data":"1ddd528f98fc8534d5bbe04248254495d50e01af0e3f021c4501ea8c5e3e6384"}
Oct 30 19:45:42.726708 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:42.726374    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ddd528f98fc8534d5bbe04248254495d50e01af0e3f021c4501ea8c5e3e6384"
Oct 30 19:45:43.634746 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:43.634706    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr"
Oct 30 19:45:43.648330 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:43.648273    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5881801f-8f93-4d5f-b303-2c89af131c12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5881801f-8f93-4d5f-b303-2c89af131c12-istio-data\") pod \"5881801f-8f93-4d5f-b303-2c89af131c12\" (UID: \"5881801f-8f93-4d5f-b303-2c89af131c12\") "
Oct 30 19:45:43.648330 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:43.648315    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5881801f-8f93-4d5f-b303-2c89af131c12-kube-api-access-kvxb5\" (UniqueName: \"kubernetes.io/projected/5881801f-8f93-4d5f-b303-2c89af131c12-kube-api-access-kvxb5\") pod \"5881801f-8f93-4d5f-b303-2c89af131c12\" (UID: \"5881801f-8f93-4d5f-b303-2c89af131c12\") "
Oct 30 19:45:43.648545 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:43.648351    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5881801f-8f93-4d5f-b303-2c89af131c12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5881801f-8f93-4d5f-b303-2c89af131c12-workload-certs\") pod \"5881801f-8f93-4d5f-b303-2c89af131c12\" (UID: \"5881801f-8f93-4d5f-b303-2c89af131c12\") "
Oct 30 19:45:43.648545 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:43.648376    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5881801f-8f93-4d5f-b303-2c89af131c12-istio-token\" (UniqueName: \"kubernetes.io/projected/5881801f-8f93-4d5f-b303-2c89af131c12-istio-token\") pod \"5881801f-8f93-4d5f-b303-2c89af131c12\" (UID: \"5881801f-8f93-4d5f-b303-2c89af131c12\") "
Oct 30 19:45:43.648545 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:43.648416    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5881801f-8f93-4d5f-b303-2c89af131c12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5881801f-8f93-4d5f-b303-2c89af131c12-istio-envoy\") pod \"5881801f-8f93-4d5f-b303-2c89af131c12\" (UID: \"5881801f-8f93-4d5f-b303-2c89af131c12\") "
Oct 30 19:45:43.648545 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:43.648448    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5881801f-8f93-4d5f-b303-2c89af131c12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5881801f-8f93-4d5f-b303-2c89af131c12-istio-podinfo\") pod \"5881801f-8f93-4d5f-b303-2c89af131c12\" (UID: \"5881801f-8f93-4d5f-b303-2c89af131c12\") "
Oct 30 19:45:43.648545 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:43.648521    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5881801f-8f93-4d5f-b303-2c89af131c12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5881801f-8f93-4d5f-b303-2c89af131c12-workload-socket\") pod \"5881801f-8f93-4d5f-b303-2c89af131c12\" (UID: \"5881801f-8f93-4d5f-b303-2c89af131c12\") "
Oct 30 19:45:43.648545 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:43.648537    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5881801f-8f93-4d5f-b303-2c89af131c12-istio-data" pod "5881801f-8f93-4d5f-b303-2c89af131c12" (UID: "5881801f-8f93-4d5f-b303-2c89af131c12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:43.648909 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:43.648573    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5881801f-8f93-4d5f-b303-2c89af131c12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5881801f-8f93-4d5f-b303-2c89af131c12-istiod-ca-cert\") pod \"5881801f-8f93-4d5f-b303-2c89af131c12\" (UID: \"5881801f-8f93-4d5f-b303-2c89af131c12\") "
Oct 30 19:45:43.648909 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:43.648618    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5881801f-8f93-4d5f-b303-2c89af131c12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5881801f-8f93-4d5f-b303-2c89af131c12-credential-socket\") pod \"5881801f-8f93-4d5f-b303-2c89af131c12\" (UID: \"5881801f-8f93-4d5f-b303-2c89af131c12\") "
Oct 30 19:45:43.648909 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:43.648672    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5881801f-8f93-4d5f-b303-2c89af131c12-workload-certs" pod "5881801f-8f93-4d5f-b303-2c89af131c12" (UID: "5881801f-8f93-4d5f-b303-2c89af131c12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:43.649052 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:43.648945    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5881801f-8f93-4d5f-b303-2c89af131c12-credential-socket" pod "5881801f-8f93-4d5f-b303-2c89af131c12" (UID: "5881801f-8f93-4d5f-b303-2c89af131c12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:43.649052 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:43.648985    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5881801f-8f93-4d5f-b303-2c89af131c12-istiod-ca-cert" pod "5881801f-8f93-4d5f-b303-2c89af131c12" (UID: "5881801f-8f93-4d5f-b303-2c89af131c12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:43.649331 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:43.649207    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5881801f-8f93-4d5f-b303-2c89af131c12-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:43.649331 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:43.649221    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5881801f-8f93-4d5f-b303-2c89af131c12-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:43.649331 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:43.649230    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5881801f-8f93-4d5f-b303-2c89af131c12-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:43.649612 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:43.649588    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5881801f-8f93-4d5f-b303-2c89af131c12-workload-socket" pod "5881801f-8f93-4d5f-b303-2c89af131c12" (UID: "5881801f-8f93-4d5f-b303-2c89af131c12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:43.650977 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:43.650954    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5881801f-8f93-4d5f-b303-2c89af131c12-istio-podinfo" pod "5881801f-8f93-4d5f-b303-2c89af131c12" (UID: "5881801f-8f93-4d5f-b303-2c89af131c12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:45:43.651084 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:43.650990    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5881801f-8f93-4d5f-b303-2c89af131c12-istio-token" pod "5881801f-8f93-4d5f-b303-2c89af131c12" (UID: "5881801f-8f93-4d5f-b303-2c89af131c12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:43.651084 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:43.651006    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5881801f-8f93-4d5f-b303-2c89af131c12-kube-api-access-kvxb5" pod "5881801f-8f93-4d5f-b303-2c89af131c12" (UID: "5881801f-8f93-4d5f-b303-2c89af131c12"). InnerVolumeSpecName "kube-api-access-kvxb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:43.651300 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:43.651284    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5881801f-8f93-4d5f-b303-2c89af131c12-istio-envoy" pod "5881801f-8f93-4d5f-b303-2c89af131c12" (UID: "5881801f-8f93-4d5f-b303-2c89af131c12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:43.736625 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:43.736590    5043 generic.go:356] "Generic (PLEG): container finished" podID="5881801f-8f93-4d5f-b303-2c89af131c12" containerID="6356573d12eb697722f991ee6d9aed500fafd31e7e2f49bc3f8521d6b5be8127" exitCode=0
Oct 30 19:45:43.737043 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:43.736663    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr"
Oct 30 19:45:43.737043 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:43.736673    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr" event={"ID":"5881801f-8f93-4d5f-b303-2c89af131c12","Type":"ContainerDied","Data":"6356573d12eb697722f991ee6d9aed500fafd31e7e2f49bc3f8521d6b5be8127"}
Oct 30 19:45:43.737043 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:43.736784    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr" event={"ID":"5881801f-8f93-4d5f-b303-2c89af131c12","Type":"ContainerDied","Data":"e06103eaa4c38ce4b0a9e173883b6860fc3e86c78b364d2312c967e33140d7d5"}
Oct 30 19:45:43.737043 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:43.736810    5043 scope.go:122] "RemoveContainer" containerID="6356573d12eb697722f991ee6d9aed500fafd31e7e2f49bc3f8521d6b5be8127"
Oct 30 19:45:43.748126 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:43.748110    5043 scope.go:122] "RemoveContainer" containerID="b4ef42071b9b00e236ae88b2f93b6d461af6502e8c788fa7c3857bcd67b0ddd9"
Oct 30 19:45:43.749876 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:43.749857    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5881801f-8f93-4d5f-b303-2c89af131c12-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:43.750078 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:43.749881    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvxb5\" (UniqueName: \"kubernetes.io/projected/5881801f-8f93-4d5f-b303-2c89af131c12-kube-api-access-kvxb5\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:43.750078 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:43.749898    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5881801f-8f93-4d5f-b303-2c89af131c12-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:43.750078 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:43.749910    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5881801f-8f93-4d5f-b303-2c89af131c12-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:43.750078 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:43.749923    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5881801f-8f93-4d5f-b303-2c89af131c12-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:43.750078 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:43.749935    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5881801f-8f93-4d5f-b303-2c89af131c12-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:43.761666 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:43.761640    5043 scope.go:122] "RemoveContainer" containerID="6356573d12eb697722f991ee6d9aed500fafd31e7e2f49bc3f8521d6b5be8127"
Oct 30 19:45:43.761999 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:45:43.761973    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6356573d12eb697722f991ee6d9aed500fafd31e7e2f49bc3f8521d6b5be8127\": container with ID starting with 6356573d12eb697722f991ee6d9aed500fafd31e7e2f49bc3f8521d6b5be8127 not found: ID does not exist" containerID="6356573d12eb697722f991ee6d9aed500fafd31e7e2f49bc3f8521d6b5be8127"
Oct 30 19:45:43.762095 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:43.762005    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6356573d12eb697722f991ee6d9aed500fafd31e7e2f49bc3f8521d6b5be8127"} err="failed to get container status \"6356573d12eb697722f991ee6d9aed500fafd31e7e2f49bc3f8521d6b5be8127\": rpc error: code = NotFound desc = could not find container \"6356573d12eb697722f991ee6d9aed500fafd31e7e2f49bc3f8521d6b5be8127\": container with ID starting with 6356573d12eb697722f991ee6d9aed500fafd31e7e2f49bc3f8521d6b5be8127 not found: ID does not exist"
Oct 30 19:45:43.762095 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:43.762029    5043 scope.go:122] "RemoveContainer" containerID="b4ef42071b9b00e236ae88b2f93b6d461af6502e8c788fa7c3857bcd67b0ddd9"
Oct 30 19:45:43.762367 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:45:43.762342    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4ef42071b9b00e236ae88b2f93b6d461af6502e8c788fa7c3857bcd67b0ddd9\": container with ID starting with b4ef42071b9b00e236ae88b2f93b6d461af6502e8c788fa7c3857bcd67b0ddd9 not found: ID does not exist" containerID="b4ef42071b9b00e236ae88b2f93b6d461af6502e8c788fa7c3857bcd67b0ddd9"
Oct 30 19:45:43.762415 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:43.762378    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4ef42071b9b00e236ae88b2f93b6d461af6502e8c788fa7c3857bcd67b0ddd9"} err="failed to get container status \"b4ef42071b9b00e236ae88b2f93b6d461af6502e8c788fa7c3857bcd67b0ddd9\": rpc error: code = NotFound desc = could not find container \"b4ef42071b9b00e236ae88b2f93b6d461af6502e8c788fa7c3857bcd67b0ddd9\": container with ID starting with b4ef42071b9b00e236ae88b2f93b6d461af6502e8c788fa7c3857bcd67b0ddd9 not found: ID does not exist"
Oct 30 19:45:43.772960 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:43.772934    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr"]
Oct 30 19:45:43.775760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:43.775726    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-976b80-openshift-default-69d4fb4fc9-6sdwr"]
Oct 30 19:45:43.867835 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:43.867799    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"]
Oct 30 19:45:43.875673 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:43.875644    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"]
Oct 30 19:45:43.881334 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:43.881299    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"]
Oct 30 19:45:44.645157 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:44.645118    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5881801f-8f93-4d5f-b303-2c89af131c12" path="/var/lib/kubelet/pods/5881801f-8f93-4d5f-b303-2c89af131c12/volumes"
Oct 30 19:45:45.284213 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:45.284161    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-47110f-7cdc4b47f5-2brw4" podUID="83c175fe-cdfd-4a54-88b6-f9a2061a11dd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:45:47.610449 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:47.610406    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46"]
Oct 30 19:45:47.610973 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:47.610663    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46" podUID="bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e" containerName="istio-proxy" containerID="cri-o://313d3de3030cb40c4cc6193cd87d37b73de4c2b7c754f231ecf326eeb0150c1f" gracePeriod=30
Oct 30 19:45:48.349251 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:48.349218    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ad96a4-x9mvs"]
Oct 30 19:45:48.349805 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:48.349786    5043 cpu_manager.go:414] "containerMap: removing container" podUID="abc3d11d-936f-41d9-a35f-adf6ab7f7909" containerName="istio-proxy"
Oct 30 19:45:48.349873 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:48.349806    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abc3d11d-936f-41d9-a35f-adf6ab7f7909" containerName="istio-proxy"
Oct 30 19:45:48.349873 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:48.349818    5043 cpu_manager.go:414] "containerMap: removing container" podUID="5881801f-8f93-4d5f-b303-2c89af131c12" containerName="istio-proxy"
Oct 30 19:45:48.349873 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:48.349823    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5881801f-8f93-4d5f-b303-2c89af131c12" containerName="istio-proxy"
Oct 30 19:45:48.349873 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:48.349845    5043 cpu_manager.go:414] "containerMap: removing container" podUID="5881801f-8f93-4d5f-b303-2c89af131c12" containerName="istio-proxy"
Oct 30 19:45:48.349873 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:48.349851    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5881801f-8f93-4d5f-b303-2c89af131c12" containerName="istio-proxy"
Oct 30 19:45:48.349873 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:48.349866    5043 cpu_manager.go:414] "containerMap: removing container" podUID="ceac8507-3941-431a-9880-2fb98e79e257" containerName="payload-processing"
Oct 30 19:45:48.349873 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:48.349872    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceac8507-3941-431a-9880-2fb98e79e257" containerName="payload-processing"
Oct 30 19:45:48.350120 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:48.349880    5043 cpu_manager.go:414] "containerMap: removing container" podUID="7af56765-aede-4480-8b11-83646e0f5284" containerName="istio-proxy"
Oct 30 19:45:48.350120 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:48.349885    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7af56765-aede-4480-8b11-83646e0f5284" containerName="istio-proxy"
Oct 30 19:45:48.350120 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:48.349898    5043 cpu_manager.go:414] "containerMap: removing container" podUID="30f3ac96-552e-40b3-8f49-eea750ba8584" containerName="revoke-keys"
Oct 30 19:45:48.350120 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:48.349903    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30f3ac96-552e-40b3-8f49-eea750ba8584" containerName="revoke-keys"
Oct 30 19:45:48.350120 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:48.349985    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="5881801f-8f93-4d5f-b303-2c89af131c12" containerName="istio-proxy"
Oct 30 19:45:48.350120 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:48.349995    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceac8507-3941-431a-9880-2fb98e79e257" containerName="payload-processing"
Oct 30 19:45:48.350120 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:48.350002    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="7af56765-aede-4480-8b11-83646e0f5284" containerName="istio-proxy"
Oct 30 19:45:48.350120 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:48.350007    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="abc3d11d-936f-41d9-a35f-adf6ab7f7909" containerName="istio-proxy"
Oct 30 19:45:48.350120 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:48.350013    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="30f3ac96-552e-40b3-8f49-eea750ba8584" containerName="revoke-keys"
Oct 30 19:45:48.350120 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:48.350020    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="abc3d11d-936f-41d9-a35f-adf6ab7f7909" containerName="istio-proxy"
Oct 30 19:45:48.354967 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:48.354946    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ad96a4-x9mvs"
Oct 30 19:45:48.366468 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:48.366443    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:45:48.366629 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:48.366540    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:45:48.370926 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:48.370902    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ad96a4-x9mvs"]
Oct 30 19:45:48.397405 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:48.397381    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37983ea9-5e58-443b-9386-4d0e3a3f68b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ad96a4-x9mvs\" (UID: \"37983ea9-5e58-443b-9386-4d0e3a3f68b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ad96a4-x9mvs"
Oct 30 19:45:48.397657 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:48.397423    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37983ea9-5e58-443b-9386-4d0e3a3f68b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ad96a4-x9mvs\" (UID: \"37983ea9-5e58-443b-9386-4d0e3a3f68b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ad96a4-x9mvs"
Oct 30 19:45:48.397657 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:48.397612    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snwms\" (UniqueName: \"kubernetes.io/projected/37983ea9-5e58-443b-9386-4d0e3a3f68b0-kube-api-access-snwms\") pod \"maas-api-revoke-keys-e2e-worker-w4-ad96a4-x9mvs\" (UID: \"37983ea9-5e58-443b-9386-4d0e3a3f68b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ad96a4-x9mvs"
Oct 30 19:45:48.448140 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:48.448109    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:45:48.448140 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:48.448130    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:45:48.499029 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:48.498996    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snwms\" (UniqueName: \"kubernetes.io/projected/37983ea9-5e58-443b-9386-4d0e3a3f68b0-kube-api-access-snwms\") pod \"maas-api-revoke-keys-e2e-worker-w4-ad96a4-x9mvs\" (UID: \"37983ea9-5e58-443b-9386-4d0e3a3f68b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ad96a4-x9mvs"
Oct 30 19:45:48.499214 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:48.499085    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37983ea9-5e58-443b-9386-4d0e3a3f68b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ad96a4-x9mvs\" (UID: \"37983ea9-5e58-443b-9386-4d0e3a3f68b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ad96a4-x9mvs"
Oct 30 19:45:48.499214 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:48.499116    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37983ea9-5e58-443b-9386-4d0e3a3f68b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ad96a4-x9mvs\" (UID: \"37983ea9-5e58-443b-9386-4d0e3a3f68b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ad96a4-x9mvs"
Oct 30 19:45:48.499817 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:48.499794    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37983ea9-5e58-443b-9386-4d0e3a3f68b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ad96a4-x9mvs\" (UID: \"37983ea9-5e58-443b-9386-4d0e3a3f68b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ad96a4-x9mvs"
Oct 30 19:45:48.501693 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:48.501664    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37983ea9-5e58-443b-9386-4d0e3a3f68b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ad96a4-x9mvs\" (UID: \"37983ea9-5e58-443b-9386-4d0e3a3f68b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ad96a4-x9mvs"
Oct 30 19:45:48.508479 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:48.508457    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snwms\" (UniqueName: \"kubernetes.io/projected/37983ea9-5e58-443b-9386-4d0e3a3f68b0-kube-api-access-snwms\") pod \"maas-api-revoke-keys-e2e-worker-w4-ad96a4-x9mvs\" (UID: \"37983ea9-5e58-443b-9386-4d0e3a3f68b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ad96a4-x9mvs"
Oct 30 19:45:48.670345 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:48.670271    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ad96a4-x9mvs"
Oct 30 19:45:48.953067 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:48.953039    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ad96a4-x9mvs"]
Oct 30 19:45:48.955415 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:45:48.955384    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37983ea9_5e58_443b_9386_4d0e3a3f68b0.slice/crio-a28edc07bfc920c6e68cd6a6665fb4a956291bf2e151759529745737688009ad WatchSource:0}: Error finding container a28edc07bfc920c6e68cd6a6665fb4a956291bf2e151759529745737688009ad: Status 404 returned error can't find the container with id a28edc07bfc920c6e68cd6a6665fb4a956291bf2e151759529745737688009ad
Oct 30 19:45:49.265683 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:49.265653    5043 patch_prober.go:29] interesting pod/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:45:49.266014 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:49.265707    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46" podUID="bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:45:49.361622 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:49.361588    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1366788f-zl2qk"]
Oct 30 19:45:49.362202 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:49.362178    5043 cpu_manager.go:414] "containerMap: removing container" podUID="abc3d11d-936f-41d9-a35f-adf6ab7f7909" containerName="istio-proxy"
Oct 30 19:45:49.362338 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:49.362206    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abc3d11d-936f-41d9-a35f-adf6ab7f7909" containerName="istio-proxy"
Oct 30 19:45:49.362436 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:49.362397    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="5881801f-8f93-4d5f-b303-2c89af131c12" containerName="istio-proxy"
Oct 30 19:45:49.366096 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:49.366077    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1366788f-zl2qk"
Oct 30 19:45:49.382837 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:49.382813    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1366788f-zl2qk"]
Oct 30 19:45:49.410246 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:49.410206    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fca56991-a8aa-48ff-b1ad-99443549fa8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1366788f-zl2qk\" (UID: \"fca56991-a8aa-48ff-b1ad-99443549fa8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1366788f-zl2qk"
Oct 30 19:45:49.410423 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:49.410255    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhkhb\" (UniqueName: \"kubernetes.io/projected/fca56991-a8aa-48ff-b1ad-99443549fa8c-kube-api-access-bhkhb\") pod \"maas-api-revoke-sub-1366788f-zl2qk\" (UID: \"fca56991-a8aa-48ff-b1ad-99443549fa8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1366788f-zl2qk"
Oct 30 19:45:49.410423 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:49.410379    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fca56991-a8aa-48ff-b1ad-99443549fa8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1366788f-zl2qk\" (UID: \"fca56991-a8aa-48ff-b1ad-99443549fa8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1366788f-zl2qk"
Oct 30 19:45:49.511195 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:49.511161    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fca56991-a8aa-48ff-b1ad-99443549fa8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1366788f-zl2qk\" (UID: \"fca56991-a8aa-48ff-b1ad-99443549fa8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1366788f-zl2qk"
Oct 30 19:45:49.511386 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:49.511216    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhkhb\" (UniqueName: \"kubernetes.io/projected/fca56991-a8aa-48ff-b1ad-99443549fa8c-kube-api-access-bhkhb\") pod \"maas-api-revoke-sub-1366788f-zl2qk\" (UID: \"fca56991-a8aa-48ff-b1ad-99443549fa8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1366788f-zl2qk"
Oct 30 19:45:49.511466 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:49.511387    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fca56991-a8aa-48ff-b1ad-99443549fa8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1366788f-zl2qk\" (UID: \"fca56991-a8aa-48ff-b1ad-99443549fa8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1366788f-zl2qk"
Oct 30 19:45:49.512214 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:49.512186    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fca56991-a8aa-48ff-b1ad-99443549fa8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1366788f-zl2qk\" (UID: \"fca56991-a8aa-48ff-b1ad-99443549fa8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1366788f-zl2qk"
Oct 30 19:45:49.513715 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:49.513697    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fca56991-a8aa-48ff-b1ad-99443549fa8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1366788f-zl2qk\" (UID: \"fca56991-a8aa-48ff-b1ad-99443549fa8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1366788f-zl2qk"
Oct 30 19:45:49.528910 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:49.528851    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhkhb\" (UniqueName: \"kubernetes.io/projected/fca56991-a8aa-48ff-b1ad-99443549fa8c-kube-api-access-bhkhb\") pod \"maas-api-revoke-sub-1366788f-zl2qk\" (UID: \"fca56991-a8aa-48ff-b1ad-99443549fa8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1366788f-zl2qk"
Oct 30 19:45:49.681151 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:49.681116    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1366788f-zl2qk"
Oct 30 19:45:49.789296 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:49.789191    5043 generic.go:356] "Generic (PLEG): container finished" podID="37983ea9-5e58-443b-9386-4d0e3a3f68b0" containerID="71601132b4a59dd38806cf63339ac3449f4068b3a3cb15c62348e1497370bb64" exitCode=0
Oct 30 19:45:49.789675 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:49.789415    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ad96a4-x9mvs" event={"ID":"37983ea9-5e58-443b-9386-4d0e3a3f68b0","Type":"ContainerDied","Data":"71601132b4a59dd38806cf63339ac3449f4068b3a3cb15c62348e1497370bb64"}
Oct 30 19:45:49.789675 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:49.789475    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ad96a4-x9mvs" event={"ID":"37983ea9-5e58-443b-9386-4d0e3a3f68b0","Type":"ContainerStarted","Data":"a28edc07bfc920c6e68cd6a6665fb4a956291bf2e151759529745737688009ad"}
Oct 30 19:45:49.973449 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:49.973419    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1366788f-zl2qk"]
Oct 30 19:45:49.974942 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:45:49.974898    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfca56991_a8aa_48ff_b1ad_99443549fa8c.slice/crio-a697a7a9ace720c36bfc323ca9e6d67725fc8a8bedb1fa34d5b6a58e9a9ae9b6 WatchSource:0}: Error finding container a697a7a9ace720c36bfc323ca9e6d67725fc8a8bedb1fa34d5b6a58e9a9ae9b6: Status 404 returned error can't find the container with id a697a7a9ace720c36bfc323ca9e6d67725fc8a8bedb1fa34d5b6a58e9a9ae9b6
Oct 30 19:45:50.803841 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:50.803789    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1366788f-zl2qk" event={"ID":"fca56991-a8aa-48ff-b1ad-99443549fa8c","Type":"ContainerStarted","Data":"5491144ca6d2af6af53ed2fe79781586dec14f5ad981271d97b6001ff78c5920"}
Oct 30 19:45:50.804331 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:50.803857    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1366788f-zl2qk" event={"ID":"fca56991-a8aa-48ff-b1ad-99443549fa8c","Type":"ContainerStarted","Data":"a697a7a9ace720c36bfc323ca9e6d67725fc8a8bedb1fa34d5b6a58e9a9ae9b6"}
Oct 30 19:45:50.821214 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:50.821090    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1366788f-zl2qk" podStartSLOduration=1.8210722430000001 podStartE2EDuration="1.821072243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:50.820612476 +0000 UTC m=+1770.331672450" watchObservedRunningTime="2026-10-30 19:45:50.821072243 +0000 UTC m=+1770.332132210"
Oct 30 19:45:50.963705 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:50.963676    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ad96a4-x9mvs"
Oct 30 19:45:51.029025 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:51.028996    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37983ea9-5e58-443b-9386-4d0e3a3f68b0-kube-api-access-snwms\" (UniqueName: \"kubernetes.io/projected/37983ea9-5e58-443b-9386-4d0e3a3f68b0-kube-api-access-snwms\") pod \"37983ea9-5e58-443b-9386-4d0e3a3f68b0\" (UID: \"37983ea9-5e58-443b-9386-4d0e3a3f68b0\") "
Oct 30 19:45:51.029223 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:51.029036    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37983ea9-5e58-443b-9386-4d0e3a3f68b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37983ea9-5e58-443b-9386-4d0e3a3f68b0-maas-api-openshift-ca\") pod \"37983ea9-5e58-443b-9386-4d0e3a3f68b0\" (UID: \"37983ea9-5e58-443b-9386-4d0e3a3f68b0\") "
Oct 30 19:45:51.029223 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:51.029196    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37983ea9-5e58-443b-9386-4d0e3a3f68b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37983ea9-5e58-443b-9386-4d0e3a3f68b0-maas-api-xks-ca\") pod \"37983ea9-5e58-443b-9386-4d0e3a3f68b0\" (UID: \"37983ea9-5e58-443b-9386-4d0e3a3f68b0\") "
Oct 30 19:45:51.029501 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:51.029465    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37983ea9-5e58-443b-9386-4d0e3a3f68b0-maas-api-openshift-ca" pod "37983ea9-5e58-443b-9386-4d0e3a3f68b0" (UID: "37983ea9-5e58-443b-9386-4d0e3a3f68b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:51.029684 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:51.029666    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37983ea9-5e58-443b-9386-4d0e3a3f68b0-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:51.031587 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:51.031539    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37983ea9-5e58-443b-9386-4d0e3a3f68b0-maas-api-xks-ca" pod "37983ea9-5e58-443b-9386-4d0e3a3f68b0" (UID: "37983ea9-5e58-443b-9386-4d0e3a3f68b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:45:51.033308 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:51.033280    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37983ea9-5e58-443b-9386-4d0e3a3f68b0-kube-api-access-snwms" pod "37983ea9-5e58-443b-9386-4d0e3a3f68b0" (UID: "37983ea9-5e58-443b-9386-4d0e3a3f68b0"). InnerVolumeSpecName "kube-api-access-snwms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:51.130527 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:51.130492    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37983ea9-5e58-443b-9386-4d0e3a3f68b0-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:51.130527 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:51.130528    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snwms\" (UniqueName: \"kubernetes.io/projected/37983ea9-5e58-443b-9386-4d0e3a3f68b0-kube-api-access-snwms\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:51.825865 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:51.825785    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ad96a4-x9mvs"
Oct 30 19:45:51.826341 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:51.825884    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ad96a4-x9mvs" event={"ID":"37983ea9-5e58-443b-9386-4d0e3a3f68b0","Type":"ContainerDied","Data":"a28edc07bfc920c6e68cd6a6665fb4a956291bf2e151759529745737688009ad"}
Oct 30 19:45:51.826341 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:51.825982    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a28edc07bfc920c6e68cd6a6665fb4a956291bf2e151759529745737688009ad"
Oct 30 19:45:51.828212 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:51.828183    5043 generic.go:356] "Generic (PLEG): container finished" podID="fca56991-a8aa-48ff-b1ad-99443549fa8c" containerID="5491144ca6d2af6af53ed2fe79781586dec14f5ad981271d97b6001ff78c5920" exitCode=0
Oct 30 19:45:51.828361 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:51.828268    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1366788f-zl2qk" event={"ID":"fca56991-a8aa-48ff-b1ad-99443549fa8c","Type":"ContainerDied","Data":"5491144ca6d2af6af53ed2fe79781586dec14f5ad981271d97b6001ff78c5920"}
Oct 30 19:45:52.844299 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:52.844247    5043 generic.go:356] "Generic (PLEG): container finished" podID="bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e" containerID="313d3de3030cb40c4cc6193cd87d37b73de4c2b7c754f231ecf326eeb0150c1f" exitCode=0
Oct 30 19:45:52.844827 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:52.844319    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46" event={"ID":"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e","Type":"ContainerDied","Data":"313d3de3030cb40c4cc6193cd87d37b73de4c2b7c754f231ecf326eeb0150c1f"}
Oct 30 19:45:52.844827 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:52.844401    5043 scope.go:122] "RemoveContainer" containerID="6c16073da015e4edf494cce7836e9ace33fde259c9c9db2a491e02bdbfce47f1"
Oct 30 19:45:52.913694 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:52.913662    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46"
Oct 30 19:45:52.952776 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:52.951072    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-istiod-ca-cert\") pod \"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e\" (UID: \"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e\") "
Oct 30 19:45:52.952776 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:52.951120    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-istio-data\") pod \"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e\" (UID: \"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e\") "
Oct 30 19:45:52.952776 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:52.951159    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-kube-api-access-ntnlc\" (UniqueName: \"kubernetes.io/projected/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-kube-api-access-ntnlc\") pod \"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e\" (UID: \"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e\") "
Oct 30 19:45:52.952776 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:52.951228    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-workload-socket\") pod \"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e\" (UID: \"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e\") "
Oct 30 19:45:52.952776 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:52.951257    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-workload-certs\") pod \"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e\" (UID: \"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e\") "
Oct 30 19:45:52.952776 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:52.951308    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-credential-socket\") pod \"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e\" (UID: \"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e\") "
Oct 30 19:45:52.952776 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:52.951361    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-istio-podinfo\") pod \"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e\" (UID: \"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e\") "
Oct 30 19:45:52.952776 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:52.951384    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-istio-token\" (UniqueName: \"kubernetes.io/projected/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-istio-token\") pod \"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e\" (UID: \"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e\") "
Oct 30 19:45:52.952776 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:52.951421    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-istio-envoy\") pod \"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e\" (UID: \"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e\") "
Oct 30 19:45:52.953776 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:52.953541    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-workload-socket" pod "bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e" (UID: "bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:52.953776 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:52.953555    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-istio-data" pod "bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e" (UID: "bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:52.958275 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:52.954408    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-credential-socket" pod "bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e" (UID: "bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:52.958275 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:52.954820    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-istiod-ca-cert" pod "bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e" (UID: "bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:52.958275 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:52.954945    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-workload-certs" pod "bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e" (UID: "bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:52.958704 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:52.958670    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-istio-envoy" pod "bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e" (UID: "bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:52.973315 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:52.973247    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-kube-api-access-ntnlc" pod "bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e" (UID: "bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e"). InnerVolumeSpecName "kube-api-access-ntnlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:52.973690 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:52.973653    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-istio-podinfo" pod "bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e" (UID: "bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:45:52.973969 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:52.973938    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-istio-token" pod "bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e" (UID: "bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:53.052550 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:53.052517    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:53.052550 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:53.052552    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:53.052779 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:53.052566    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:53.052779 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:53.052580    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:53.052779 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:53.052594    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:53.052779 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:53.052608    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntnlc\" (UniqueName: \"kubernetes.io/projected/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-kube-api-access-ntnlc\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:53.052779 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:53.052622    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:53.052779 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:53.052636    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:53.052779 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:53.052650    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:53.055351 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:53.055327    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1366788f-zl2qk"
Oct 30 19:45:53.154083 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:53.153990    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fca56991-a8aa-48ff-b1ad-99443549fa8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fca56991-a8aa-48ff-b1ad-99443549fa8c-maas-api-openshift-ca\") pod \"fca56991-a8aa-48ff-b1ad-99443549fa8c\" (UID: \"fca56991-a8aa-48ff-b1ad-99443549fa8c\") "
Oct 30 19:45:53.154274 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:53.154137    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fca56991-a8aa-48ff-b1ad-99443549fa8c-kube-api-access-bhkhb\" (UniqueName: \"kubernetes.io/projected/fca56991-a8aa-48ff-b1ad-99443549fa8c-kube-api-access-bhkhb\") pod \"fca56991-a8aa-48ff-b1ad-99443549fa8c\" (UID: \"fca56991-a8aa-48ff-b1ad-99443549fa8c\") "
Oct 30 19:45:53.154274 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:53.154233    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fca56991-a8aa-48ff-b1ad-99443549fa8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fca56991-a8aa-48ff-b1ad-99443549fa8c-maas-api-xks-ca\") pod \"fca56991-a8aa-48ff-b1ad-99443549fa8c\" (UID: \"fca56991-a8aa-48ff-b1ad-99443549fa8c\") "
Oct 30 19:45:53.154446 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:53.154421    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fca56991-a8aa-48ff-b1ad-99443549fa8c-maas-api-openshift-ca" pod "fca56991-a8aa-48ff-b1ad-99443549fa8c" (UID: "fca56991-a8aa-48ff-b1ad-99443549fa8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:53.154657 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:53.154632    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fca56991-a8aa-48ff-b1ad-99443549fa8c-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:53.156540 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:53.156502    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fca56991-a8aa-48ff-b1ad-99443549fa8c-kube-api-access-bhkhb" pod "fca56991-a8aa-48ff-b1ad-99443549fa8c" (UID: "fca56991-a8aa-48ff-b1ad-99443549fa8c"). InnerVolumeSpecName "kube-api-access-bhkhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:53.156658 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:53.156589    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fca56991-a8aa-48ff-b1ad-99443549fa8c-maas-api-xks-ca" pod "fca56991-a8aa-48ff-b1ad-99443549fa8c" (UID: "fca56991-a8aa-48ff-b1ad-99443549fa8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:45:53.255288 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:53.255250    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fca56991-a8aa-48ff-b1ad-99443549fa8c-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:53.255288 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:53.255284    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhkhb\" (UniqueName: \"kubernetes.io/projected/fca56991-a8aa-48ff-b1ad-99443549fa8c-kube-api-access-bhkhb\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:53.854105 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:53.854074    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1366788f-zl2qk"
Oct 30 19:45:53.854542 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:53.854113    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1366788f-zl2qk" event={"ID":"fca56991-a8aa-48ff-b1ad-99443549fa8c","Type":"ContainerDied","Data":"a697a7a9ace720c36bfc323ca9e6d67725fc8a8bedb1fa34d5b6a58e9a9ae9b6"}
Oct 30 19:45:53.854542 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:53.854165    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a697a7a9ace720c36bfc323ca9e6d67725fc8a8bedb1fa34d5b6a58e9a9ae9b6"
Oct 30 19:45:53.855863 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:53.855835    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46"
Oct 30 19:45:53.856001 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:53.855866    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46" event={"ID":"bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e","Type":"ContainerDied","Data":"eb265abed4fa4a38e43190bd6c7bc60442559f1d3d691caffe26a5f834a1a5be"}
Oct 30 19:45:53.856001 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:53.855916    5043 scope.go:122] "RemoveContainer" containerID="313d3de3030cb40c4cc6193cd87d37b73de4c2b7c754f231ecf326eeb0150c1f"
Oct 30 19:45:53.900334 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:53.900264    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46"]
Oct 30 19:45:53.909203 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:53.909175    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-jtp46"]
Oct 30 19:45:54.647375 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:54.647342    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e" path="/var/lib/kubelet/pods/bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e/volumes"
Oct 30 19:45:54.977781 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:45:54.977718    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:45:55.284019 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:55.283929    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-47110f-7cdc4b47f5-2brw4" podUID="83c175fe-cdfd-4a54-88b6-f9a2061a11dd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:45:58.497240 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:58.497205    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ad96a4-746867c7d7-lnlgf"]
Oct 30 19:45:58.892275 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:58.892150    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main" containerID="cri-o://f52c942d70e971961ba7977e5fec2a34186a403576cc02998112ac2e906e9c87" gracePeriod=45
Oct 30 19:45:58.898782 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:58.898726    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" containerID="cri-o://3ab96f67f367f04f33b41a5298cb888f97257541d7ca8328a7c46a7dd1821d83" gracePeriod=45
Oct 30 19:45:58.910157 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:58.910120    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main" containerID="cri-o://da526e8eeae35cfed2aeeb58503f31186452edf143681be97da5b2f3bfcda7b9" gracePeriod=45
Oct 30 19:45:59.195070 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.195033    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:45:59.198358 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.198329    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:45:59.230074 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.230022    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:45:59.323968 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.323937    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e898751-6f2a-43cf-9d48-f03358f1de63-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e898751-6f2a-43cf-9d48-f03358f1de63-model-cache\") pod \"0e898751-6f2a-43cf-9d48-f03358f1de63\" (UID: \"0e898751-6f2a-43cf-9d48-f03358f1de63\") "
Oct 30 19:45:59.324216 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.323984    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c54b39a-1658-4bd9-aad0-5ec529126201-tls-certs\" (UniqueName: \"kubernetes.io/secret/7c54b39a-1658-4bd9-aad0-5ec529126201-tls-certs\") pod \"7c54b39a-1658-4bd9-aad0-5ec529126201\" (UID: \"7c54b39a-1658-4bd9-aad0-5ec529126201\") "
Oct 30 19:45:59.324216 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.324012    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/583dd7dc-b160-468a-8ebe-781bead5e989-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/583dd7dc-b160-468a-8ebe-781bead5e989-model-cache\") pod \"583dd7dc-b160-468a-8ebe-781bead5e989\" (UID: \"583dd7dc-b160-468a-8ebe-781bead5e989\") "
Oct 30 19:45:59.324216 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.324056    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/583dd7dc-b160-468a-8ebe-781bead5e989-dshm\" (UniqueName: \"kubernetes.io/empty-dir/583dd7dc-b160-468a-8ebe-781bead5e989-dshm\") pod \"583dd7dc-b160-468a-8ebe-781bead5e989\" (UID: \"583dd7dc-b160-468a-8ebe-781bead5e989\") "
Oct 30 19:45:59.324216 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.324118    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c54b39a-1658-4bd9-aad0-5ec529126201-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c54b39a-1658-4bd9-aad0-5ec529126201-dshm\") pod \"7c54b39a-1658-4bd9-aad0-5ec529126201\" (UID: \"7c54b39a-1658-4bd9-aad0-5ec529126201\") "
Oct 30 19:45:59.324216 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.324156    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/583dd7dc-b160-468a-8ebe-781bead5e989-home\" (UniqueName: \"kubernetes.io/empty-dir/583dd7dc-b160-468a-8ebe-781bead5e989-home\") pod \"583dd7dc-b160-468a-8ebe-781bead5e989\" (UID: \"583dd7dc-b160-468a-8ebe-781bead5e989\") "
Oct 30 19:45:59.324216 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.324187    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c54b39a-1658-4bd9-aad0-5ec529126201-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c54b39a-1658-4bd9-aad0-5ec529126201-model-cache\") pod \"7c54b39a-1658-4bd9-aad0-5ec529126201\" (UID: \"7c54b39a-1658-4bd9-aad0-5ec529126201\") "
Oct 30 19:45:59.324596 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.324227    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/583dd7dc-b160-468a-8ebe-781bead5e989-tls-certs\" (UniqueName: \"kubernetes.io/secret/583dd7dc-b160-468a-8ebe-781bead5e989-tls-certs\") pod \"583dd7dc-b160-468a-8ebe-781bead5e989\" (UID: \"583dd7dc-b160-468a-8ebe-781bead5e989\") "
Oct 30 19:45:59.324596 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.324269    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e898751-6f2a-43cf-9d48-f03358f1de63-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e898751-6f2a-43cf-9d48-f03358f1de63-dshm\") pod \"0e898751-6f2a-43cf-9d48-f03358f1de63\" (UID: \"0e898751-6f2a-43cf-9d48-f03358f1de63\") "
Oct 30 19:45:59.324596 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.324268    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e898751-6f2a-43cf-9d48-f03358f1de63-model-cache" pod "0e898751-6f2a-43cf-9d48-f03358f1de63" (UID: "0e898751-6f2a-43cf-9d48-f03358f1de63"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:59.324596 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.324306    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/583dd7dc-b160-468a-8ebe-781bead5e989-kube-api-access-5lsp9\" (UniqueName: \"kubernetes.io/projected/583dd7dc-b160-468a-8ebe-781bead5e989-kube-api-access-5lsp9\") pod \"583dd7dc-b160-468a-8ebe-781bead5e989\" (UID: \"583dd7dc-b160-468a-8ebe-781bead5e989\") "
Oct 30 19:45:59.324596 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.324312    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/583dd7dc-b160-468a-8ebe-781bead5e989-model-cache" pod "583dd7dc-b160-468a-8ebe-781bead5e989" (UID: "583dd7dc-b160-468a-8ebe-781bead5e989"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:59.324596 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.324349    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e898751-6f2a-43cf-9d48-f03358f1de63-home\" (UniqueName: \"kubernetes.io/empty-dir/0e898751-6f2a-43cf-9d48-f03358f1de63-home\") pod \"0e898751-6f2a-43cf-9d48-f03358f1de63\" (UID: \"0e898751-6f2a-43cf-9d48-f03358f1de63\") "
Oct 30 19:45:59.324596 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.324393    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/583dd7dc-b160-468a-8ebe-781bead5e989-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/583dd7dc-b160-468a-8ebe-781bead5e989-tmp-dir\") pod \"583dd7dc-b160-468a-8ebe-781bead5e989\" (UID: \"583dd7dc-b160-468a-8ebe-781bead5e989\") "
Oct 30 19:45:59.324596 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.324422    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c54b39a-1658-4bd9-aad0-5ec529126201-home\" (UniqueName: \"kubernetes.io/empty-dir/7c54b39a-1658-4bd9-aad0-5ec529126201-home\") pod \"7c54b39a-1658-4bd9-aad0-5ec529126201\" (UID: \"7c54b39a-1658-4bd9-aad0-5ec529126201\") "
Oct 30 19:45:59.324596 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.324469    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c54b39a-1658-4bd9-aad0-5ec529126201-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c54b39a-1658-4bd9-aad0-5ec529126201-tmp-dir\") pod \"7c54b39a-1658-4bd9-aad0-5ec529126201\" (UID: \"7c54b39a-1658-4bd9-aad0-5ec529126201\") "
Oct 30 19:45:59.324596 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.324503    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e898751-6f2a-43cf-9d48-f03358f1de63-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e898751-6f2a-43cf-9d48-f03358f1de63-tmp-dir\") pod \"0e898751-6f2a-43cf-9d48-f03358f1de63\" (UID: \"0e898751-6f2a-43cf-9d48-f03358f1de63\") "
Oct 30 19:45:59.324596 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.324510    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c54b39a-1658-4bd9-aad0-5ec529126201-model-cache" pod "7c54b39a-1658-4bd9-aad0-5ec529126201" (UID: "7c54b39a-1658-4bd9-aad0-5ec529126201"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:59.324596 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.324530    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/583dd7dc-b160-468a-8ebe-781bead5e989-home" pod "583dd7dc-b160-468a-8ebe-781bead5e989" (UID: "583dd7dc-b160-468a-8ebe-781bead5e989"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:59.324596 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.324599    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e898751-6f2a-43cf-9d48-f03358f1de63-kube-api-access-sssrl\" (UniqueName: \"kubernetes.io/projected/0e898751-6f2a-43cf-9d48-f03358f1de63-kube-api-access-sssrl\") pod \"0e898751-6f2a-43cf-9d48-f03358f1de63\" (UID: \"0e898751-6f2a-43cf-9d48-f03358f1de63\") "
Oct 30 19:45:59.325611 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.324637    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c54b39a-1658-4bd9-aad0-5ec529126201-kube-api-access-r7nkm\" (UniqueName: \"kubernetes.io/projected/7c54b39a-1658-4bd9-aad0-5ec529126201-kube-api-access-r7nkm\") pod \"7c54b39a-1658-4bd9-aad0-5ec529126201\" (UID: \"7c54b39a-1658-4bd9-aad0-5ec529126201\") "
Oct 30 19:45:59.325611 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.324711    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e898751-6f2a-43cf-9d48-f03358f1de63-tls-certs\" (UniqueName: \"kubernetes.io/secret/0e898751-6f2a-43cf-9d48-f03358f1de63-tls-certs\") pod \"0e898751-6f2a-43cf-9d48-f03358f1de63\" (UID: \"0e898751-6f2a-43cf-9d48-f03358f1de63\") "
Oct 30 19:45:59.325611 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.324724    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c54b39a-1658-4bd9-aad0-5ec529126201-home" pod "7c54b39a-1658-4bd9-aad0-5ec529126201" (UID: "7c54b39a-1658-4bd9-aad0-5ec529126201"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:59.325611 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.324752    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e898751-6f2a-43cf-9d48-f03358f1de63-home" pod "0e898751-6f2a-43cf-9d48-f03358f1de63" (UID: "0e898751-6f2a-43cf-9d48-f03358f1de63"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:59.325611 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.325007    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e898751-6f2a-43cf-9d48-f03358f1de63-tmp-dir" pod "0e898751-6f2a-43cf-9d48-f03358f1de63" (UID: "0e898751-6f2a-43cf-9d48-f03358f1de63"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:59.325611 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.325110    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/583dd7dc-b160-468a-8ebe-781bead5e989-tmp-dir" pod "583dd7dc-b160-468a-8ebe-781bead5e989" (UID: "583dd7dc-b160-468a-8ebe-781bead5e989"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:59.325611 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.325180    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c54b39a-1658-4bd9-aad0-5ec529126201-tmp-dir" pod "7c54b39a-1658-4bd9-aad0-5ec529126201" (UID: "7c54b39a-1658-4bd9-aad0-5ec529126201"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:59.325611 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.325257    5043 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e898751-6f2a-43cf-9d48-f03358f1de63-model-cache\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:59.325611 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.325277    5043 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/583dd7dc-b160-468a-8ebe-781bead5e989-model-cache\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:59.325611 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.325292    5043 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/583dd7dc-b160-468a-8ebe-781bead5e989-home\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:59.325611 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.325305    5043 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c54b39a-1658-4bd9-aad0-5ec529126201-model-cache\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:59.325611 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.325318    5043 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e898751-6f2a-43cf-9d48-f03358f1de63-home\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:59.325611 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.325333    5043 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/583dd7dc-b160-468a-8ebe-781bead5e989-tmp-dir\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:59.325611 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.325345    5043 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c54b39a-1658-4bd9-aad0-5ec529126201-home\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:59.325611 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.325356    5043 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e898751-6f2a-43cf-9d48-f03358f1de63-tmp-dir\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:59.327953 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.327672    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/583dd7dc-b160-468a-8ebe-781bead5e989-kube-api-access-5lsp9" pod "583dd7dc-b160-468a-8ebe-781bead5e989" (UID: "583dd7dc-b160-468a-8ebe-781bead5e989"). InnerVolumeSpecName "kube-api-access-5lsp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:59.328240 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.328216    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c54b39a-1658-4bd9-aad0-5ec529126201-kube-api-access-r7nkm" pod "7c54b39a-1658-4bd9-aad0-5ec529126201" (UID: "7c54b39a-1658-4bd9-aad0-5ec529126201"). InnerVolumeSpecName "kube-api-access-r7nkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:59.328386 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.328362    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e898751-6f2a-43cf-9d48-f03358f1de63-dshm" pod "0e898751-6f2a-43cf-9d48-f03358f1de63" (UID: "0e898751-6f2a-43cf-9d48-f03358f1de63"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:59.330000 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.329974    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/583dd7dc-b160-468a-8ebe-781bead5e989-dshm" pod "583dd7dc-b160-468a-8ebe-781bead5e989" (UID: "583dd7dc-b160-468a-8ebe-781bead5e989"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:59.330000 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.329984    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c54b39a-1658-4bd9-aad0-5ec529126201-dshm" pod "7c54b39a-1658-4bd9-aad0-5ec529126201" (UID: "7c54b39a-1658-4bd9-aad0-5ec529126201"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:59.330000 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.329990    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e898751-6f2a-43cf-9d48-f03358f1de63-kube-api-access-sssrl" pod "0e898751-6f2a-43cf-9d48-f03358f1de63" (UID: "0e898751-6f2a-43cf-9d48-f03358f1de63"). InnerVolumeSpecName "kube-api-access-sssrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:59.330174 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.330005    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c54b39a-1658-4bd9-aad0-5ec529126201-tls-certs" pod "7c54b39a-1658-4bd9-aad0-5ec529126201" (UID: "7c54b39a-1658-4bd9-aad0-5ec529126201"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:45:59.330263 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.330248    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e898751-6f2a-43cf-9d48-f03358f1de63-tls-certs" pod "0e898751-6f2a-43cf-9d48-f03358f1de63" (UID: "0e898751-6f2a-43cf-9d48-f03358f1de63"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:45:59.331299 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.331277    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/583dd7dc-b160-468a-8ebe-781bead5e989-tls-certs" pod "583dd7dc-b160-468a-8ebe-781bead5e989" (UID: "583dd7dc-b160-468a-8ebe-781bead5e989"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:45:59.335642 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.335619    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-803dc14d-jqdgz"]
Oct 30 19:45:59.336090 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.336077    5043 cpu_manager.go:414] "containerMap: removing container" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main"
Oct 30 19:45:59.336147 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.336092    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main"
Oct 30 19:45:59.336147 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.336112    5043 cpu_manager.go:414] "containerMap: removing container" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main"
Oct 30 19:45:59.336147 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.336118    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main"
Oct 30 19:45:59.336147 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.336132    5043 cpu_manager.go:414] "containerMap: removing container" podUID="bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e" containerName="istio-proxy"
Oct 30 19:45:59.336147 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.336137    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e" containerName="istio-proxy"
Oct 30 19:45:59.336147 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.336146    5043 cpu_manager.go:414] "containerMap: removing container" podUID="37983ea9-5e58-443b-9386-4d0e3a3f68b0" containerName="revoke-keys"
Oct 30 19:45:59.336342 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.336151    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37983ea9-5e58-443b-9386-4d0e3a3f68b0" containerName="revoke-keys"
Oct 30 19:45:59.336342 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.336159    5043 cpu_manager.go:414] "containerMap: removing container" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main"
Oct 30 19:45:59.336342 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.336164    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main"
Oct 30 19:45:59.336342 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.336171    5043 cpu_manager.go:414] "containerMap: removing container" podUID="fca56991-a8aa-48ff-b1ad-99443549fa8c" containerName="revoke-keys"
Oct 30 19:45:59.336342 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.336175    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fca56991-a8aa-48ff-b1ad-99443549fa8c" containerName="revoke-keys"
Oct 30 19:45:59.336342 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.336182    5043 cpu_manager.go:414] "containerMap: removing container" podUID="bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e" containerName="istio-proxy"
Oct 30 19:45:59.336342 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.336187    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e" containerName="istio-proxy"
Oct 30 19:45:59.336342 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.336262    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="37983ea9-5e58-443b-9386-4d0e3a3f68b0" containerName="revoke-keys"
Oct 30 19:45:59.336342 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.336270    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main"
Oct 30 19:45:59.336342 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.336277    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="fca56991-a8aa-48ff-b1ad-99443549fa8c" containerName="revoke-keys"
Oct 30 19:45:59.336342 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.336283    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e" containerName="istio-proxy"
Oct 30 19:45:59.336342 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.336305    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerName="main"
Oct 30 19:45:59.336342 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.336312    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerName="main"
Oct 30 19:45:59.340980 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.340959    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-803dc14d-jqdgz"
Oct 30 19:45:59.343385 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.343366    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5109a2f-kzgdn"]
Oct 30 19:45:59.343929 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.343913    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd2e9ea0-1574-4cd8-9e79-206d8f06dc8e" containerName="istio-proxy"
Oct 30 19:45:59.348972 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.348435    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:45:59.348972 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.348510    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:45:59.348972 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.348550    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:45:59.358672 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.358646    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-803dc14d-jqdgz"]
Oct 30 19:45:59.358831 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.358783    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5109a2f-kzgdn"
Oct 30 19:45:59.364143 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.364076    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5109a2f-kzgdn"]
Oct 30 19:45:59.426479 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.426401    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aef173f-b001-4494-89df-65d747c56b31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-803dc14d-jqdgz\" (UID: \"5aef173f-b001-4494-89df-65d747c56b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-803dc14d-jqdgz"
Oct 30 19:45:59.426479 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.426443    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aef173f-b001-4494-89df-65d747c56b31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-803dc14d-jqdgz\" (UID: \"5aef173f-b001-4494-89df-65d747c56b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-803dc14d-jqdgz"
Oct 30 19:45:59.426673 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.426544    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f9df\" (UniqueName: \"kubernetes.io/projected/5aef173f-b001-4494-89df-65d747c56b31-kube-api-access-7f9df\") pod \"maas-api-revoke-sub-803dc14d-jqdgz\" (UID: \"5aef173f-b001-4494-89df-65d747c56b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-803dc14d-jqdgz"
Oct 30 19:45:59.426673 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.426574    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c78899fb-4833-45cb-bb84-8c15ae19be7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5109a2f-kzgdn\" (UID: \"c78899fb-4833-45cb-bb84-8c15ae19be7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5109a2f-kzgdn"
Oct 30 19:45:59.426789 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.426683    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhrk9\" (UniqueName: \"kubernetes.io/projected/c78899fb-4833-45cb-bb84-8c15ae19be7c-kube-api-access-bhrk9\") pod \"maas-api-revoke-sub-a5109a2f-kzgdn\" (UID: \"c78899fb-4833-45cb-bb84-8c15ae19be7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5109a2f-kzgdn"
Oct 30 19:45:59.426789 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.426724    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c78899fb-4833-45cb-bb84-8c15ae19be7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5109a2f-kzgdn\" (UID: \"c78899fb-4833-45cb-bb84-8c15ae19be7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5109a2f-kzgdn"
Oct 30 19:45:59.426891 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.426877    5043 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c54b39a-1658-4bd9-aad0-5ec529126201-tls-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:59.426928 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.426893    5043 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/583dd7dc-b160-468a-8ebe-781bead5e989-dshm\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:59.426928 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.426901    5043 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c54b39a-1658-4bd9-aad0-5ec529126201-dshm\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:59.426928 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.426909    5043 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/583dd7dc-b160-468a-8ebe-781bead5e989-tls-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:59.426928 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.426919    5043 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e898751-6f2a-43cf-9d48-f03358f1de63-dshm\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:59.427064 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.426928    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lsp9\" (UniqueName: \"kubernetes.io/projected/583dd7dc-b160-468a-8ebe-781bead5e989-kube-api-access-5lsp9\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:59.427064 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.426937    5043 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c54b39a-1658-4bd9-aad0-5ec529126201-tmp-dir\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:59.427064 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.426945    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sssrl\" (UniqueName: \"kubernetes.io/projected/0e898751-6f2a-43cf-9d48-f03358f1de63-kube-api-access-sssrl\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:59.427064 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.426953    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7nkm\" (UniqueName: \"kubernetes.io/projected/7c54b39a-1658-4bd9-aad0-5ec529126201-kube-api-access-r7nkm\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:59.427064 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.426962    5043 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e898751-6f2a-43cf-9d48-f03358f1de63-tls-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:59.446311 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.446287    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:45:59.528109 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.528075    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aef173f-b001-4494-89df-65d747c56b31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-803dc14d-jqdgz\" (UID: \"5aef173f-b001-4494-89df-65d747c56b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-803dc14d-jqdgz"
Oct 30 19:45:59.528578 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.528134    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f9df\" (UniqueName: \"kubernetes.io/projected/5aef173f-b001-4494-89df-65d747c56b31-kube-api-access-7f9df\") pod \"maas-api-revoke-sub-803dc14d-jqdgz\" (UID: \"5aef173f-b001-4494-89df-65d747c56b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-803dc14d-jqdgz"
Oct 30 19:45:59.528578 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.528156    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c78899fb-4833-45cb-bb84-8c15ae19be7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5109a2f-kzgdn\" (UID: \"c78899fb-4833-45cb-bb84-8c15ae19be7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5109a2f-kzgdn"
Oct 30 19:45:59.528578 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.528201    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhrk9\" (UniqueName: \"kubernetes.io/projected/c78899fb-4833-45cb-bb84-8c15ae19be7c-kube-api-access-bhrk9\") pod \"maas-api-revoke-sub-a5109a2f-kzgdn\" (UID: \"c78899fb-4833-45cb-bb84-8c15ae19be7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5109a2f-kzgdn"
Oct 30 19:45:59.528578 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.528232    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c78899fb-4833-45cb-bb84-8c15ae19be7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5109a2f-kzgdn\" (UID: \"c78899fb-4833-45cb-bb84-8c15ae19be7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5109a2f-kzgdn"
Oct 30 19:45:59.528578 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.528414    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aef173f-b001-4494-89df-65d747c56b31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-803dc14d-jqdgz\" (UID: \"5aef173f-b001-4494-89df-65d747c56b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-803dc14d-jqdgz"
Oct 30 19:45:59.529100 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.529057    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c78899fb-4833-45cb-bb84-8c15ae19be7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5109a2f-kzgdn\" (UID: \"c78899fb-4833-45cb-bb84-8c15ae19be7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5109a2f-kzgdn"
Oct 30 19:45:59.529249 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.529208    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aef173f-b001-4494-89df-65d747c56b31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-803dc14d-jqdgz\" (UID: \"5aef173f-b001-4494-89df-65d747c56b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-803dc14d-jqdgz"
Oct 30 19:45:59.530666 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.530647    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aef173f-b001-4494-89df-65d747c56b31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-803dc14d-jqdgz\" (UID: \"5aef173f-b001-4494-89df-65d747c56b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-803dc14d-jqdgz"
Oct 30 19:45:59.530777 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.530698    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c78899fb-4833-45cb-bb84-8c15ae19be7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5109a2f-kzgdn\" (UID: \"c78899fb-4833-45cb-bb84-8c15ae19be7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5109a2f-kzgdn"
Oct 30 19:45:59.538880 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.538854    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhrk9\" (UniqueName: \"kubernetes.io/projected/c78899fb-4833-45cb-bb84-8c15ae19be7c-kube-api-access-bhrk9\") pod \"maas-api-revoke-sub-a5109a2f-kzgdn\" (UID: \"c78899fb-4833-45cb-bb84-8c15ae19be7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5109a2f-kzgdn"
Oct 30 19:45:59.544623 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.544598    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f9df\" (UniqueName: \"kubernetes.io/projected/5aef173f-b001-4494-89df-65d747c56b31-kube-api-access-7f9df\") pod \"maas-api-revoke-sub-803dc14d-jqdgz\" (UID: \"5aef173f-b001-4494-89df-65d747c56b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-803dc14d-jqdgz"
Oct 30 19:45:59.664884 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.664850    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-803dc14d-jqdgz"
Oct 30 19:45:59.679146 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.679124    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5109a2f-kzgdn"
Oct 30 19:45:59.908295 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.908252    5043 generic.go:356] "Generic (PLEG): container finished" podID="7c54b39a-1658-4bd9-aad0-5ec529126201" containerID="f52c942d70e971961ba7977e5fec2a34186a403576cc02998112ac2e906e9c87" exitCode=0
Oct 30 19:45:59.908494 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.908311    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"
Oct 30 19:45:59.908494 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.908377    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" event={"ID":"7c54b39a-1658-4bd9-aad0-5ec529126201","Type":"ContainerDied","Data":"f52c942d70e971961ba7977e5fec2a34186a403576cc02998112ac2e906e9c87"}
Oct 30 19:45:59.908494 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.908414    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp" event={"ID":"7c54b39a-1658-4bd9-aad0-5ec529126201","Type":"ContainerDied","Data":"a19e2dd6e3f8ddc23c864d2efb4c822f20085623b9b803b8d4ef146c79191113"}
Oct 30 19:45:59.908494 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.908436    5043 scope.go:122] "RemoveContainer" containerID="f52c942d70e971961ba7977e5fec2a34186a403576cc02998112ac2e906e9c87"
Oct 30 19:45:59.914486 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.914452    5043 generic.go:356] "Generic (PLEG): container finished" podID="0e898751-6f2a-43cf-9d48-f03358f1de63" containerID="da526e8eeae35cfed2aeeb58503f31186452edf143681be97da5b2f3bfcda7b9" exitCode=0
Oct 30 19:45:59.914621 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.914504    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"
Oct 30 19:45:59.914621 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.914523    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" event={"ID":"0e898751-6f2a-43cf-9d48-f03358f1de63","Type":"ContainerDied","Data":"da526e8eeae35cfed2aeeb58503f31186452edf143681be97da5b2f3bfcda7b9"}
Oct 30 19:45:59.914621 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.914569    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx" event={"ID":"0e898751-6f2a-43cf-9d48-f03358f1de63","Type":"ContainerDied","Data":"2ea9cce2661012f9d85e91547bca30862e68243dfd6b7386d9b70e397922edd6"}
Oct 30 19:45:59.921984 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.921959    5043 generic.go:356] "Generic (PLEG): container finished" podID="583dd7dc-b160-468a-8ebe-781bead5e989" containerID="3ab96f67f367f04f33b41a5298cb888f97257541d7ca8328a7c46a7dd1821d83" exitCode=0
Oct 30 19:45:59.922113 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.922011    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"
Oct 30 19:45:59.922113 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.922039    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" event={"ID":"583dd7dc-b160-468a-8ebe-781bead5e989","Type":"ContainerDied","Data":"3ab96f67f367f04f33b41a5298cb888f97257541d7ca8328a7c46a7dd1821d83"}
Oct 30 19:45:59.922113 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.922075    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" event={"ID":"583dd7dc-b160-468a-8ebe-781bead5e989","Type":"ContainerDied","Data":"5a86424bfb321ba734bd024945cce8194549fd6cb9273025cc4eab5735d6c587"}
Oct 30 19:45:59.939962 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.939938    5043 scope.go:122] "RemoveContainer" containerID="f52c942d70e971961ba7977e5fec2a34186a403576cc02998112ac2e906e9c87"
Oct 30 19:45:59.940340 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:45:59.940316    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f52c942d70e971961ba7977e5fec2a34186a403576cc02998112ac2e906e9c87\": container with ID starting with f52c942d70e971961ba7977e5fec2a34186a403576cc02998112ac2e906e9c87 not found: ID does not exist" containerID="f52c942d70e971961ba7977e5fec2a34186a403576cc02998112ac2e906e9c87"
Oct 30 19:45:59.940418 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.940356    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f52c942d70e971961ba7977e5fec2a34186a403576cc02998112ac2e906e9c87"} err="failed to get container status \"f52c942d70e971961ba7977e5fec2a34186a403576cc02998112ac2e906e9c87\": rpc error: code = NotFound desc = could not find container \"f52c942d70e971961ba7977e5fec2a34186a403576cc02998112ac2e906e9c87\": container with ID starting with f52c942d70e971961ba7977e5fec2a34186a403576cc02998112ac2e906e9c87 not found: ID does not exist"
Oct 30 19:45:59.940418 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.940379    5043 scope.go:122] "RemoveContainer" containerID="da526e8eeae35cfed2aeeb58503f31186452edf143681be97da5b2f3bfcda7b9"
Oct 30 19:45:59.951370 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.951346    5043 scope.go:122] "RemoveContainer" containerID="da526e8eeae35cfed2aeeb58503f31186452edf143681be97da5b2f3bfcda7b9"
Oct 30 19:45:59.951848 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:45:59.951818    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da526e8eeae35cfed2aeeb58503f31186452edf143681be97da5b2f3bfcda7b9\": container with ID starting with da526e8eeae35cfed2aeeb58503f31186452edf143681be97da5b2f3bfcda7b9 not found: ID does not exist" containerID="da526e8eeae35cfed2aeeb58503f31186452edf143681be97da5b2f3bfcda7b9"
Oct 30 19:45:59.952046 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.952016    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da526e8eeae35cfed2aeeb58503f31186452edf143681be97da5b2f3bfcda7b9"} err="failed to get container status \"da526e8eeae35cfed2aeeb58503f31186452edf143681be97da5b2f3bfcda7b9\": rpc error: code = NotFound desc = could not find container \"da526e8eeae35cfed2aeeb58503f31186452edf143681be97da5b2f3bfcda7b9\": container with ID starting with da526e8eeae35cfed2aeeb58503f31186452edf143681be97da5b2f3bfcda7b9 not found: ID does not exist"
Oct 30 19:45:59.952046 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.952045    5043 scope.go:122] "RemoveContainer" containerID="3ab96f67f367f04f33b41a5298cb888f97257541d7ca8328a7c46a7dd1821d83"
Oct 30 19:45:59.957010 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.956977    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"]
Oct 30 19:45:59.962691 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.962668    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-803dc14d-jqdgz"]
Oct 30 19:45:59.965928 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.965906    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-ce69f3/distinct-w3-ce69f3-kserve-6f4fd6db8-f59dp"]
Oct 30 19:45:59.968945 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.968924    5043 scope.go:122] "RemoveContainer" containerID="3ab96f67f367f04f33b41a5298cb888f97257541d7ca8328a7c46a7dd1821d83"
Oct 30 19:45:59.969289 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:45:59.969256    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ab96f67f367f04f33b41a5298cb888f97257541d7ca8328a7c46a7dd1821d83\": container with ID starting with 3ab96f67f367f04f33b41a5298cb888f97257541d7ca8328a7c46a7dd1821d83 not found: ID does not exist" containerID="3ab96f67f367f04f33b41a5298cb888f97257541d7ca8328a7c46a7dd1821d83"
Oct 30 19:45:59.969366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.969301    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ab96f67f367f04f33b41a5298cb888f97257541d7ca8328a7c46a7dd1821d83"} err="failed to get container status \"3ab96f67f367f04f33b41a5298cb888f97257541d7ca8328a7c46a7dd1821d83\": rpc error: code = NotFound desc = could not find container \"3ab96f67f367f04f33b41a5298cb888f97257541d7ca8328a7c46a7dd1821d83\": container with ID starting with 3ab96f67f367f04f33b41a5298cb888f97257541d7ca8328a7c46a7dd1821d83 not found: ID does not exist"
Oct 30 19:45:59.970430 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:45:59.970409    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aef173f_b001_4494_89df_65d747c56b31.slice/crio-07539ad0eca9630680c5a15f54faebdefb4cb9d3ee9c32616b4f4f6ad15090fd WatchSource:0}: Error finding container 07539ad0eca9630680c5a15f54faebdefb4cb9d3ee9c32616b4f4f6ad15090fd: Status 404 returned error can't find the container with id 07539ad0eca9630680c5a15f54faebdefb4cb9d3ee9c32616b4f4f6ad15090fd
Oct 30 19:45:59.981162 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.981138    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5109a2f-kzgdn"]
Oct 30 19:45:59.986491 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.986440    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"]
Oct 30 19:45:59.991338 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:45:59.991305    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-ce69f3/simulator-w3-ce69f3-kserve-587ffdc8c6-t45tx"]
Oct 30 19:46:00.003477 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:00.003440    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"]
Oct 30 19:46:00.015397 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:00.015372    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf"]
Oct 30 19:46:00.129374 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:00.129316    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/premium-w3-ce69f3-kserve-7bd85bd65-5z7vf" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:46:00.648248 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:00.648213    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e898751-6f2a-43cf-9d48-f03358f1de63" path="/var/lib/kubelet/pods/0e898751-6f2a-43cf-9d48-f03358f1de63/volumes"
Oct 30 19:46:00.648770 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:00.648665    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="583dd7dc-b160-468a-8ebe-781bead5e989" path="/var/lib/kubelet/pods/583dd7dc-b160-468a-8ebe-781bead5e989/volumes"
Oct 30 19:46:00.649101 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:00.649081    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c54b39a-1658-4bd9-aad0-5ec529126201" path="/var/lib/kubelet/pods/7c54b39a-1658-4bd9-aad0-5ec529126201/volumes"
Oct 30 19:46:00.693843 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:00.693820    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-47110f-7cdc4b47f5-2brw4"
Oct 30 19:46:00.742230 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:00.742194    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83c175fe-cdfd-4a54-88b6-f9a2061a11dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83c175fe-cdfd-4a54-88b6-f9a2061a11dd-plugins-config-volume\") pod \"83c175fe-cdfd-4a54-88b6-f9a2061a11dd\" (UID: \"83c175fe-cdfd-4a54-88b6-f9a2061a11dd\") "
Oct 30 19:46:00.742407 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:00.742249    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83c175fe-cdfd-4a54-88b6-f9a2061a11dd-kube-api-access-c6fz9\" (UniqueName: \"kubernetes.io/projected/83c175fe-cdfd-4a54-88b6-f9a2061a11dd-kube-api-access-c6fz9\") pod \"83c175fe-cdfd-4a54-88b6-f9a2061a11dd\" (UID: \"83c175fe-cdfd-4a54-88b6-f9a2061a11dd\") "
Oct 30 19:46:00.742648 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:00.742621    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83c175fe-cdfd-4a54-88b6-f9a2061a11dd-plugins-config-volume" pod "83c175fe-cdfd-4a54-88b6-f9a2061a11dd" (UID: "83c175fe-cdfd-4a54-88b6-f9a2061a11dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:00.744627 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:00.744605    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c175fe-cdfd-4a54-88b6-f9a2061a11dd-kube-api-access-c6fz9" pod "83c175fe-cdfd-4a54-88b6-f9a2061a11dd" (UID: "83c175fe-cdfd-4a54-88b6-f9a2061a11dd"). InnerVolumeSpecName "kube-api-access-c6fz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:00.843812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:00.843720    5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83c175fe-cdfd-4a54-88b6-f9a2061a11dd-plugins-config-volume\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:00.843812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:00.843760    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6fz9\" (UniqueName: \"kubernetes.io/projected/83c175fe-cdfd-4a54-88b6-f9a2061a11dd-kube-api-access-c6fz9\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:00.931468 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:00.931432    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5109a2f-kzgdn" event={"ID":"c78899fb-4833-45cb-bb84-8c15ae19be7c","Type":"ContainerStarted","Data":"895e1a6c43d723545fac739cbd49ae8df23565c184bd7515a5abe314406e0b87"}
Oct 30 19:46:00.931615 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:00.931497    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5109a2f-kzgdn" event={"ID":"c78899fb-4833-45cb-bb84-8c15ae19be7c","Type":"ContainerStarted","Data":"aeebd2b3b4114547d4e44650c8e2058c9e177d8319ae6a6d3a7c4c55168b96a5"}
Oct 30 19:46:00.935937 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:00.935910    5043 generic.go:356] "Generic (PLEG): container finished" podID="5aef173f-b001-4494-89df-65d747c56b31" containerID="3e07fd0d14ca89a2cffbc25631e0458a8adff022a03abbd4a90d979961e0b1b3" exitCode=0
Oct 30 19:46:00.936088 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:00.935950    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-803dc14d-jqdgz" event={"ID":"5aef173f-b001-4494-89df-65d747c56b31","Type":"ContainerDied","Data":"3e07fd0d14ca89a2cffbc25631e0458a8adff022a03abbd4a90d979961e0b1b3"}
Oct 30 19:46:00.936088 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:00.935996    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-803dc14d-jqdgz" event={"ID":"5aef173f-b001-4494-89df-65d747c56b31","Type":"ContainerStarted","Data":"07539ad0eca9630680c5a15f54faebdefb4cb9d3ee9c32616b4f4f6ad15090fd"}
Oct 30 19:46:00.938364 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:00.938336    5043 generic.go:356] "Generic (PLEG): container finished" podID="83c175fe-cdfd-4a54-88b6-f9a2061a11dd" containerID="6f7549fab0ce5ce26f5fd1a06edaaacc6b005037e60cac8a11744b52ceea452f" exitCode=0
Oct 30 19:46:00.938508 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:00.938381    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-47110f-7cdc4b47f5-2brw4"
Oct 30 19:46:00.938508 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:00.938419    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-47110f-7cdc4b47f5-2brw4" event={"ID":"83c175fe-cdfd-4a54-88b6-f9a2061a11dd","Type":"ContainerDied","Data":"6f7549fab0ce5ce26f5fd1a06edaaacc6b005037e60cac8a11744b52ceea452f"}
Oct 30 19:46:00.938508 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:00.938485    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-47110f-7cdc4b47f5-2brw4" event={"ID":"83c175fe-cdfd-4a54-88b6-f9a2061a11dd","Type":"ContainerDied","Data":"bdc71f7a89cc77a81ae152c1b3a878324f0176c3a076d4d13030e9c2abbd1f2c"}
Oct 30 19:46:00.938670 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:00.938512    5043 scope.go:122] "RemoveContainer" containerID="6f7549fab0ce5ce26f5fd1a06edaaacc6b005037e60cac8a11744b52ceea452f"
Oct 30 19:46:00.949236 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:00.949218    5043 scope.go:122] "RemoveContainer" containerID="6f7549fab0ce5ce26f5fd1a06edaaacc6b005037e60cac8a11744b52ceea452f"
Oct 30 19:46:00.949560 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:46:00.949541    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f7549fab0ce5ce26f5fd1a06edaaacc6b005037e60cac8a11744b52ceea452f\": container with ID starting with 6f7549fab0ce5ce26f5fd1a06edaaacc6b005037e60cac8a11744b52ceea452f not found: ID does not exist" containerID="6f7549fab0ce5ce26f5fd1a06edaaacc6b005037e60cac8a11744b52ceea452f"
Oct 30 19:46:00.949602 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:00.949569    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f7549fab0ce5ce26f5fd1a06edaaacc6b005037e60cac8a11744b52ceea452f"} err="failed to get container status \"6f7549fab0ce5ce26f5fd1a06edaaacc6b005037e60cac8a11744b52ceea452f\": rpc error: code = NotFound desc = could not find container \"6f7549fab0ce5ce26f5fd1a06edaaacc6b005037e60cac8a11744b52ceea452f\": container with ID starting with 6f7549fab0ce5ce26f5fd1a06edaaacc6b005037e60cac8a11744b52ceea452f not found: ID does not exist"
Oct 30 19:46:00.957118 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:00.957067    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5109a2f-kzgdn" podStartSLOduration=1.957049896 podStartE2EDuration="1.957049896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:00.952685441 +0000 UTC m=+1780.463745406" watchObservedRunningTime="2026-10-30 19:46:00.957049896 +0000 UTC m=+1780.468109866"
Oct 30 19:46:00.998132 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:00.998104    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-47110f-7cdc4b47f5-2brw4"]
Oct 30 19:46:01.012311 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:01.012277    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-47110f-7cdc4b47f5-2brw4"]
Oct 30 19:46:01.904140 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:01.904106    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ce69f3-f6rwg"]
Oct 30 19:46:01.907013 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:01.906993    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ce69f3-f6rwg"]
Oct 30 19:46:01.948878 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:01.948840    5043 generic.go:356] "Generic (PLEG): container finished" podID="c78899fb-4833-45cb-bb84-8c15ae19be7c" containerID="895e1a6c43d723545fac739cbd49ae8df23565c184bd7515a5abe314406e0b87" exitCode=0
Oct 30 19:46:01.949056 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:01.948932    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5109a2f-kzgdn" event={"ID":"c78899fb-4833-45cb-bb84-8c15ae19be7c","Type":"ContainerDied","Data":"895e1a6c43d723545fac739cbd49ae8df23565c184bd7515a5abe314406e0b87"}
Oct 30 19:46:02.114221 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:02.114196    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-803dc14d-jqdgz"
Oct 30 19:46:02.157722 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:02.157640    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5aef173f-b001-4494-89df-65d747c56b31-kube-api-access-7f9df\" (UniqueName: \"kubernetes.io/projected/5aef173f-b001-4494-89df-65d747c56b31-kube-api-access-7f9df\") pod \"5aef173f-b001-4494-89df-65d747c56b31\" (UID: \"5aef173f-b001-4494-89df-65d747c56b31\") "
Oct 30 19:46:02.157918 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:02.157762    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5aef173f-b001-4494-89df-65d747c56b31-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aef173f-b001-4494-89df-65d747c56b31-maas-api-xks-ca\") pod \"5aef173f-b001-4494-89df-65d747c56b31\" (UID: \"5aef173f-b001-4494-89df-65d747c56b31\") "
Oct 30 19:46:02.157918 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:02.157834    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5aef173f-b001-4494-89df-65d747c56b31-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aef173f-b001-4494-89df-65d747c56b31-maas-api-openshift-ca\") pod \"5aef173f-b001-4494-89df-65d747c56b31\" (UID: \"5aef173f-b001-4494-89df-65d747c56b31\") "
Oct 30 19:46:02.158259 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:02.158233    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5aef173f-b001-4494-89df-65d747c56b31-maas-api-openshift-ca" pod "5aef173f-b001-4494-89df-65d747c56b31" (UID: "5aef173f-b001-4494-89df-65d747c56b31"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:02.160780 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:02.160163    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5aef173f-b001-4494-89df-65d747c56b31-maas-api-xks-ca" pod "5aef173f-b001-4494-89df-65d747c56b31" (UID: "5aef173f-b001-4494-89df-65d747c56b31"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:46:02.161213 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:02.161185    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aef173f-b001-4494-89df-65d747c56b31-kube-api-access-7f9df" pod "5aef173f-b001-4494-89df-65d747c56b31" (UID: "5aef173f-b001-4494-89df-65d747c56b31"). InnerVolumeSpecName "kube-api-access-7f9df". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:02.258975 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:02.258919    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aef173f-b001-4494-89df-65d747c56b31-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:02.258975 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:02.258963    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aef173f-b001-4494-89df-65d747c56b31-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:02.258975 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:02.258973    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f9df\" (UniqueName: \"kubernetes.io/projected/5aef173f-b001-4494-89df-65d747c56b31-kube-api-access-7f9df\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:02.644753 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:02.644714    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30f3ac96-552e-40b3-8f49-eea750ba8584" path="/var/lib/kubelet/pods/30f3ac96-552e-40b3-8f49-eea750ba8584/volumes"
Oct 30 19:46:02.645173 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:02.645156    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83c175fe-cdfd-4a54-88b6-f9a2061a11dd" path="/var/lib/kubelet/pods/83c175fe-cdfd-4a54-88b6-f9a2061a11dd/volumes"
Oct 30 19:46:02.959568 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:02.959498    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-803dc14d-jqdgz"
Oct 30 19:46:02.959568 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:02.959528    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-803dc14d-jqdgz" event={"ID":"5aef173f-b001-4494-89df-65d747c56b31","Type":"ContainerDied","Data":"07539ad0eca9630680c5a15f54faebdefb4cb9d3ee9c32616b4f4f6ad15090fd"}
Oct 30 19:46:02.960176 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:02.959603    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07539ad0eca9630680c5a15f54faebdefb4cb9d3ee9c32616b4f4f6ad15090fd"
Oct 30 19:46:03.138315 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:03.138284    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5109a2f-kzgdn"
Oct 30 19:46:03.270234 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:03.270158    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c78899fb-4833-45cb-bb84-8c15ae19be7c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c78899fb-4833-45cb-bb84-8c15ae19be7c-maas-api-openshift-ca\") pod \"c78899fb-4833-45cb-bb84-8c15ae19be7c\" (UID: \"c78899fb-4833-45cb-bb84-8c15ae19be7c\") "
Oct 30 19:46:03.270234 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:03.270224    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c78899fb-4833-45cb-bb84-8c15ae19be7c-kube-api-access-bhrk9\" (UniqueName: \"kubernetes.io/projected/c78899fb-4833-45cb-bb84-8c15ae19be7c-kube-api-access-bhrk9\") pod \"c78899fb-4833-45cb-bb84-8c15ae19be7c\" (UID: \"c78899fb-4833-45cb-bb84-8c15ae19be7c\") "
Oct 30 19:46:03.270499 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:03.270278    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c78899fb-4833-45cb-bb84-8c15ae19be7c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c78899fb-4833-45cb-bb84-8c15ae19be7c-maas-api-xks-ca\") pod \"c78899fb-4833-45cb-bb84-8c15ae19be7c\" (UID: \"c78899fb-4833-45cb-bb84-8c15ae19be7c\") "
Oct 30 19:46:03.270588 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:03.270550    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c78899fb-4833-45cb-bb84-8c15ae19be7c-maas-api-openshift-ca" pod "c78899fb-4833-45cb-bb84-8c15ae19be7c" (UID: "c78899fb-4833-45cb-bb84-8c15ae19be7c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:03.272449 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:03.272422    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c78899fb-4833-45cb-bb84-8c15ae19be7c-maas-api-xks-ca" pod "c78899fb-4833-45cb-bb84-8c15ae19be7c" (UID: "c78899fb-4833-45cb-bb84-8c15ae19be7c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:46:03.272598 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:03.272482    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c78899fb-4833-45cb-bb84-8c15ae19be7c-kube-api-access-bhrk9" pod "c78899fb-4833-45cb-bb84-8c15ae19be7c" (UID: "c78899fb-4833-45cb-bb84-8c15ae19be7c"). InnerVolumeSpecName "kube-api-access-bhrk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:03.371762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:03.371402    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c78899fb-4833-45cb-bb84-8c15ae19be7c-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:03.371762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:03.371447    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhrk9\" (UniqueName: \"kubernetes.io/projected/c78899fb-4833-45cb-bb84-8c15ae19be7c-kube-api-access-bhrk9\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:03.371762 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:03.371462    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c78899fb-4833-45cb-bb84-8c15ae19be7c-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:03.510761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:03.500821    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ad96a4-746867c7d7-lnlgf" podUID="33ae4933-1096-419e-81c6-defab7a8d6e0" containerName="payload-pre-processing" containerID="cri-o://d349cba6d1451d7c8755645219ae5f49b99600ee64526b13e6750669b4fb3ab1" gracePeriod=25
Oct 30 19:46:03.976774 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:03.974418    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5109a2f-kzgdn"
Oct 30 19:46:03.976774 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:03.974449    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5109a2f-kzgdn" event={"ID":"c78899fb-4833-45cb-bb84-8c15ae19be7c","Type":"ContainerDied","Data":"aeebd2b3b4114547d4e44650c8e2058c9e177d8319ae6a6d3a7c4c55168b96a5"}
Oct 30 19:46:03.976774 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:03.974514    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aeebd2b3b4114547d4e44650c8e2058c9e177d8319ae6a6d3a7c4c55168b96a5"
Oct 30 19:46:08.506282 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:08.505917    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ad96a4-746867c7d7-lnlgf" podUID="33ae4933-1096-419e-81c6-defab7a8d6e0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:46:10.757660 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:10.757615    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv"]
Oct 30 19:46:10.758174 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:10.757884    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv" podUID="63688e51-3c1c-41ed-af40-4e2aa0e98c9e" containerName="istio-proxy" containerID="cri-o://e548144cfc3d45364a127a8f8247633b91b481e7ab36993163e27edda2c15d04" gracePeriod=30
Oct 30 19:46:12.731723 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:12.731672    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ad96a4-x9mvs"]
Oct 30 19:46:12.733140 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:12.733112    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ad96a4-x9mvs"]
Oct 30 19:46:14.645577 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:14.645541    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37983ea9-5e58-443b-9386-4d0e3a3f68b0" path="/var/lib/kubelet/pods/37983ea9-5e58-443b-9386-4d0e3a3f68b0/volumes"
Oct 30 19:46:16.048585 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:16.048475    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv"
Oct 30 19:46:16.081491 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:16.081448    5043 generic.go:356] "Generic (PLEG): container finished" podID="63688e51-3c1c-41ed-af40-4e2aa0e98c9e" containerID="e548144cfc3d45364a127a8f8247633b91b481e7ab36993163e27edda2c15d04" exitCode=0
Oct 30 19:46:16.081689 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:16.081521    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv" event={"ID":"63688e51-3c1c-41ed-af40-4e2aa0e98c9e","Type":"ContainerDied","Data":"e548144cfc3d45364a127a8f8247633b91b481e7ab36993163e27edda2c15d04"}
Oct 30 19:46:16.081689 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:16.081571    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv" event={"ID":"63688e51-3c1c-41ed-af40-4e2aa0e98c9e","Type":"ContainerDied","Data":"dad9c7a20312978b5d423f67c0490a11e154b8bc971bac7371d036b5dc07e58d"}
Oct 30 19:46:16.081689 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:16.081604    5043 scope.go:122] "RemoveContainer" containerID="e548144cfc3d45364a127a8f8247633b91b481e7ab36993163e27edda2c15d04"
Oct 30 19:46:16.081907 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:16.081710    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv"
Oct 30 19:46:16.093360 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:16.093339    5043 scope.go:122] "RemoveContainer" containerID="e548144cfc3d45364a127a8f8247633b91b481e7ab36993163e27edda2c15d04"
Oct 30 19:46:16.093700 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:46:16.093682    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e548144cfc3d45364a127a8f8247633b91b481e7ab36993163e27edda2c15d04\": container with ID starting with e548144cfc3d45364a127a8f8247633b91b481e7ab36993163e27edda2c15d04 not found: ID does not exist" containerID="e548144cfc3d45364a127a8f8247633b91b481e7ab36993163e27edda2c15d04"
Oct 30 19:46:16.093790 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:16.093712    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e548144cfc3d45364a127a8f8247633b91b481e7ab36993163e27edda2c15d04"} err="failed to get container status \"e548144cfc3d45364a127a8f8247633b91b481e7ab36993163e27edda2c15d04\": rpc error: code = NotFound desc = could not find container \"e548144cfc3d45364a127a8f8247633b91b481e7ab36993163e27edda2c15d04\": container with ID starting with e548144cfc3d45364a127a8f8247633b91b481e7ab36993163e27edda2c15d04 not found: ID does not exist"
Oct 30 19:46:16.120018 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:16.119985    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-istiod-ca-cert\") pod \"63688e51-3c1c-41ed-af40-4e2aa0e98c9e\" (UID: \"63688e51-3c1c-41ed-af40-4e2aa0e98c9e\") "
Oct 30 19:46:16.120174 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:16.120031    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-istio-token\" (UniqueName: \"kubernetes.io/projected/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-istio-token\") pod \"63688e51-3c1c-41ed-af40-4e2aa0e98c9e\" (UID: \"63688e51-3c1c-41ed-af40-4e2aa0e98c9e\") "
Oct 30 19:46:16.120174 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:16.120103    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-istio-envoy\") pod \"63688e51-3c1c-41ed-af40-4e2aa0e98c9e\" (UID: \"63688e51-3c1c-41ed-af40-4e2aa0e98c9e\") "
Oct 30 19:46:16.120174 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:16.120159    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-istio-podinfo\") pod \"63688e51-3c1c-41ed-af40-4e2aa0e98c9e\" (UID: \"63688e51-3c1c-41ed-af40-4e2aa0e98c9e\") "
Oct 30 19:46:16.120339 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:16.120215    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-kube-api-access-fqr8x\" (UniqueName: \"kubernetes.io/projected/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-kube-api-access-fqr8x\") pod \"63688e51-3c1c-41ed-af40-4e2aa0e98c9e\" (UID: \"63688e51-3c1c-41ed-af40-4e2aa0e98c9e\") "
Oct 30 19:46:16.120339 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:16.120283    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-workload-certs\") pod \"63688e51-3c1c-41ed-af40-4e2aa0e98c9e\" (UID: \"63688e51-3c1c-41ed-af40-4e2aa0e98c9e\") "
Oct 30 19:46:16.120339 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:16.120308    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-credential-socket\") pod \"63688e51-3c1c-41ed-af40-4e2aa0e98c9e\" (UID: \"63688e51-3c1c-41ed-af40-4e2aa0e98c9e\") "
Oct 30 19:46:16.120494 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:16.120340    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-istio-data\") pod \"63688e51-3c1c-41ed-af40-4e2aa0e98c9e\" (UID: \"63688e51-3c1c-41ed-af40-4e2aa0e98c9e\") "
Oct 30 19:46:16.120494 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:16.120393    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-workload-socket\") pod \"63688e51-3c1c-41ed-af40-4e2aa0e98c9e\" (UID: \"63688e51-3c1c-41ed-af40-4e2aa0e98c9e\") "
Oct 30 19:46:16.120494 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:16.120436    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-istiod-ca-cert" pod "63688e51-3c1c-41ed-af40-4e2aa0e98c9e" (UID: "63688e51-3c1c-41ed-af40-4e2aa0e98c9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:16.120663 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:16.120610    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-workload-certs" pod "63688e51-3c1c-41ed-af40-4e2aa0e98c9e" (UID: "63688e51-3c1c-41ed-af40-4e2aa0e98c9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:16.120851 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:16.120820    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-workload-socket" pod "63688e51-3c1c-41ed-af40-4e2aa0e98c9e" (UID: "63688e51-3c1c-41ed-af40-4e2aa0e98c9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:16.121141 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:16.120859    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-istio-data" pod "63688e51-3c1c-41ed-af40-4e2aa0e98c9e" (UID: "63688e51-3c1c-41ed-af40-4e2aa0e98c9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:16.121141 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:16.120874    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:16.121141 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:16.120914    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:16.121141 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:16.120964    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-credential-socket" pod "63688e51-3c1c-41ed-af40-4e2aa0e98c9e" (UID: "63688e51-3c1c-41ed-af40-4e2aa0e98c9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:16.122682 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:16.122661    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-istio-envoy" pod "63688e51-3c1c-41ed-af40-4e2aa0e98c9e" (UID: "63688e51-3c1c-41ed-af40-4e2aa0e98c9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:16.122809 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:16.122660    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-istio-podinfo" pod "63688e51-3c1c-41ed-af40-4e2aa0e98c9e" (UID: "63688e51-3c1c-41ed-af40-4e2aa0e98c9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:46:16.122874 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:16.122854    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-istio-token" pod "63688e51-3c1c-41ed-af40-4e2aa0e98c9e" (UID: "63688e51-3c1c-41ed-af40-4e2aa0e98c9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:16.122874 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:16.122852    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-kube-api-access-fqr8x" pod "63688e51-3c1c-41ed-af40-4e2aa0e98c9e" (UID: "63688e51-3c1c-41ed-af40-4e2aa0e98c9e"). InnerVolumeSpecName "kube-api-access-fqr8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:16.221667 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:16.221625    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:16.221667 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:16.221664    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:16.221667 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:16.221673    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:16.221932 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:16.221682    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:16.221932 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:16.221690    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:16.221932 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:16.221699    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:16.221932 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:16.221708    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqr8x\" (UniqueName: \"kubernetes.io/projected/63688e51-3c1c-41ed-af40-4e2aa0e98c9e-kube-api-access-fqr8x\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:16.408165 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:16.408132    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv"]
Oct 30 19:46:16.414957 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:16.414929    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-kzwqv"]
Oct 30 19:46:16.643390 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:16.643311    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63688e51-3c1c-41ed-af40-4e2aa0e98c9e" path="/var/lib/kubelet/pods/63688e51-3c1c-41ed-af40-4e2aa0e98c9e/volumes"
Oct 30 19:46:18.506916 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:18.506874    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ad96a4-746867c7d7-lnlgf" podUID="33ae4933-1096-419e-81c6-defab7a8d6e0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:46:21.179173 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:21.179140    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cmw6w_76e21916-0744-4499-a0cb-64a7a44a8285/console-operator/2.log"
Oct 30 19:46:21.179832 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:21.179289    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cmw6w_76e21916-0744-4499-a0cb-64a7a44a8285/console-operator/2.log"
Oct 30 19:46:21.410105 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:21.410072    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx"]
Oct 30 19:46:21.410301 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:21.410278    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx" podUID="f1ca4728-e019-4147-b9a3-d3f662d98e77" containerName="istio-proxy" containerID="cri-o://6927cc37fe5ae78876e2ef49ba2720ba7947b71ead78bcdc4bb804a20073f013" gracePeriod=30
Oct 30 19:46:23.773054 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:23.772987    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ad96a4-746867c7d7-lnlgf"
Oct 30 19:46:23.901852 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:23.901814    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33ae4933-1096-419e-81c6-defab7a8d6e0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33ae4933-1096-419e-81c6-defab7a8d6e0-plugins-config-volume\") pod \"33ae4933-1096-419e-81c6-defab7a8d6e0\" (UID: \"33ae4933-1096-419e-81c6-defab7a8d6e0\") "
Oct 30 19:46:23.902064 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:23.901923    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33ae4933-1096-419e-81c6-defab7a8d6e0-kube-api-access-bswvb\" (UniqueName: \"kubernetes.io/projected/33ae4933-1096-419e-81c6-defab7a8d6e0-kube-api-access-bswvb\") pod \"33ae4933-1096-419e-81c6-defab7a8d6e0\" (UID: \"33ae4933-1096-419e-81c6-defab7a8d6e0\") "
Oct 30 19:46:23.902193 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:23.902169    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33ae4933-1096-419e-81c6-defab7a8d6e0-plugins-config-volume" pod "33ae4933-1096-419e-81c6-defab7a8d6e0" (UID: "33ae4933-1096-419e-81c6-defab7a8d6e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:23.902289 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:23.902275    5043 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33ae4933-1096-419e-81c6-defab7a8d6e0-plugins-config-volume\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:23.904175 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:23.904153    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33ae4933-1096-419e-81c6-defab7a8d6e0-kube-api-access-bswvb" pod "33ae4933-1096-419e-81c6-defab7a8d6e0" (UID: "33ae4933-1096-419e-81c6-defab7a8d6e0"). InnerVolumeSpecName "kube-api-access-bswvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:24.003348 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:24.003307    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bswvb\" (UniqueName: \"kubernetes.io/projected/33ae4933-1096-419e-81c6-defab7a8d6e0-kube-api-access-bswvb\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:24.167802 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:24.167709    5043 generic.go:356] "Generic (PLEG): container finished" podID="33ae4933-1096-419e-81c6-defab7a8d6e0" containerID="d349cba6d1451d7c8755645219ae5f49b99600ee64526b13e6750669b4fb3ab1" exitCode=0
Oct 30 19:46:24.167802 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:24.167778    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ad96a4-746867c7d7-lnlgf"
Oct 30 19:46:24.167802 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:24.167778    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ad96a4-746867c7d7-lnlgf" event={"ID":"33ae4933-1096-419e-81c6-defab7a8d6e0","Type":"ContainerDied","Data":"d349cba6d1451d7c8755645219ae5f49b99600ee64526b13e6750669b4fb3ab1"}
Oct 30 19:46:24.168103 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:24.167846    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ad96a4-746867c7d7-lnlgf" event={"ID":"33ae4933-1096-419e-81c6-defab7a8d6e0","Type":"ContainerDied","Data":"4ce1e5eac8049b0d49a8909a1e88cc0784c49fc44fe92813b1761e9b8e2e6d8d"}
Oct 30 19:46:24.168103 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:24.167876    5043 scope.go:122] "RemoveContainer" containerID="d349cba6d1451d7c8755645219ae5f49b99600ee64526b13e6750669b4fb3ab1"
Oct 30 19:46:24.179417 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:24.179390    5043 scope.go:122] "RemoveContainer" containerID="d349cba6d1451d7c8755645219ae5f49b99600ee64526b13e6750669b4fb3ab1"
Oct 30 19:46:24.179722 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:46:24.179702    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d349cba6d1451d7c8755645219ae5f49b99600ee64526b13e6750669b4fb3ab1\": container with ID starting with d349cba6d1451d7c8755645219ae5f49b99600ee64526b13e6750669b4fb3ab1 not found: ID does not exist" containerID="d349cba6d1451d7c8755645219ae5f49b99600ee64526b13e6750669b4fb3ab1"
Oct 30 19:46:24.179812 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:24.179731    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d349cba6d1451d7c8755645219ae5f49b99600ee64526b13e6750669b4fb3ab1"} err="failed to get container status \"d349cba6d1451d7c8755645219ae5f49b99600ee64526b13e6750669b4fb3ab1\": rpc error: code = NotFound desc = could not find container \"d349cba6d1451d7c8755645219ae5f49b99600ee64526b13e6750669b4fb3ab1\": container with ID starting with d349cba6d1451d7c8755645219ae5f49b99600ee64526b13e6750669b4fb3ab1 not found: ID does not exist"
Oct 30 19:46:24.194640 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:24.194611    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ad96a4-746867c7d7-lnlgf"]
Oct 30 19:46:24.201748 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:24.201708    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ad96a4-746867c7d7-lnlgf"]
Oct 30 19:46:24.643534 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:24.643501    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33ae4933-1096-419e-81c6-defab7a8d6e0" path="/var/lib/kubelet/pods/33ae4933-1096-419e-81c6-defab7a8d6e0/volumes"
Oct 30 19:46:24.978874 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:46:24.978838    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:46:25.019255 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:25.019220    5043 scope.go:122] "RemoveContainer" containerID="51d20eb0846e85304fb37875a43e06632930dcbfb742dcc4a2ba4e1986c85b34"
Oct 30 19:46:25.030795 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:25.030772    5043 scope.go:122] "RemoveContainer" containerID="5478b7d677c61fc946a0dd6efdfd3928e3e7ceba65a0285d403565d98d8b38a7"
Oct 30 19:46:25.039760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:25.039727    5043 scope.go:122] "RemoveContainer" containerID="e00cba2e4fd3b7c639bf8502e73385762b9fc059b8f79b4c1ebaa33f06dd908b"
Oct 30 19:46:25.048823 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:25.048800    5043 scope.go:122] "RemoveContainer" containerID="92851e774ff23239be9c38c1d09f854a89149e87eab58b7555226c2bbd0fdbdf"
Oct 30 19:46:25.062050 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:25.062026    5043 scope.go:122] "RemoveContainer" containerID="f7174cc773f830b1388a2a84feb355e4448d37b651fd88e6f63fccc37a870165"
Oct 30 19:46:25.319672 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:25.319581    5043 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:46:25.319672 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:25.319649    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx" podUID="f1ca4728-e019-4147-b9a3-d3f662d98e77" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:46:26.773760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:26.772941    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx"
Oct 30 19:46:26.836214 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:26.836183    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1ca4728-e019-4147-b9a3-d3f662d98e77-istio-token\" (UniqueName: \"kubernetes.io/projected/f1ca4728-e019-4147-b9a3-d3f662d98e77-istio-token\") pod \"f1ca4728-e019-4147-b9a3-d3f662d98e77\" (UID: \"f1ca4728-e019-4147-b9a3-d3f662d98e77\") "
Oct 30 19:46:26.836399 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:26.836222    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1ca4728-e019-4147-b9a3-d3f662d98e77-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1ca4728-e019-4147-b9a3-d3f662d98e77-istio-envoy\") pod \"f1ca4728-e019-4147-b9a3-d3f662d98e77\" (UID: \"f1ca4728-e019-4147-b9a3-d3f662d98e77\") "
Oct 30 19:46:26.836399 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:26.836261    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1ca4728-e019-4147-b9a3-d3f662d98e77-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1ca4728-e019-4147-b9a3-d3f662d98e77-istio-data\") pod \"f1ca4728-e019-4147-b9a3-d3f662d98e77\" (UID: \"f1ca4728-e019-4147-b9a3-d3f662d98e77\") "
Oct 30 19:46:26.836399 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:26.836310    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1ca4728-e019-4147-b9a3-d3f662d98e77-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1ca4728-e019-4147-b9a3-d3f662d98e77-workload-certs\") pod \"f1ca4728-e019-4147-b9a3-d3f662d98e77\" (UID: \"f1ca4728-e019-4147-b9a3-d3f662d98e77\") "
Oct 30 19:46:26.836399 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:26.836363    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1ca4728-e019-4147-b9a3-d3f662d98e77-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1ca4728-e019-4147-b9a3-d3f662d98e77-istio-podinfo\") pod \"f1ca4728-e019-4147-b9a3-d3f662d98e77\" (UID: \"f1ca4728-e019-4147-b9a3-d3f662d98e77\") "
Oct 30 19:46:26.836541 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:26.836397    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1ca4728-e019-4147-b9a3-d3f662d98e77-kube-api-access-92nbc\" (UniqueName: \"kubernetes.io/projected/f1ca4728-e019-4147-b9a3-d3f662d98e77-kube-api-access-92nbc\") pod \"f1ca4728-e019-4147-b9a3-d3f662d98e77\" (UID: \"f1ca4728-e019-4147-b9a3-d3f662d98e77\") "
Oct 30 19:46:26.836541 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:26.836434    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1ca4728-e019-4147-b9a3-d3f662d98e77-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1ca4728-e019-4147-b9a3-d3f662d98e77-istiod-ca-cert\") pod \"f1ca4728-e019-4147-b9a3-d3f662d98e77\" (UID: \"f1ca4728-e019-4147-b9a3-d3f662d98e77\") "
Oct 30 19:46:26.836541 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:26.836476    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1ca4728-e019-4147-b9a3-d3f662d98e77-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ca4728-e019-4147-b9a3-d3f662d98e77-workload-socket\") pod \"f1ca4728-e019-4147-b9a3-d3f662d98e77\" (UID: \"f1ca4728-e019-4147-b9a3-d3f662d98e77\") "
Oct 30 19:46:26.836541 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:26.836511    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1ca4728-e019-4147-b9a3-d3f662d98e77-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ca4728-e019-4147-b9a3-d3f662d98e77-credential-socket\") pod \"f1ca4728-e019-4147-b9a3-d3f662d98e77\" (UID: \"f1ca4728-e019-4147-b9a3-d3f662d98e77\") "
Oct 30 19:46:26.837090 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:26.836847    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1ca4728-e019-4147-b9a3-d3f662d98e77-workload-certs" pod "f1ca4728-e019-4147-b9a3-d3f662d98e77" (UID: "f1ca4728-e019-4147-b9a3-d3f662d98e77"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:26.837090 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:26.836885    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1ca4728-e019-4147-b9a3-d3f662d98e77-istio-data" pod "f1ca4728-e019-4147-b9a3-d3f662d98e77" (UID: "f1ca4728-e019-4147-b9a3-d3f662d98e77"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:26.837090 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:26.837050    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1ca4728-e019-4147-b9a3-d3f662d98e77-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:26.837090 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:26.837060    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1ca4728-e019-4147-b9a3-d3f662d98e77-workload-socket" pod "f1ca4728-e019-4147-b9a3-d3f662d98e77" (UID: "f1ca4728-e019-4147-b9a3-d3f662d98e77"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:26.837090 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:26.837069    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1ca4728-e019-4147-b9a3-d3f662d98e77-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:26.837640 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:26.837286    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1ca4728-e019-4147-b9a3-d3f662d98e77-credential-socket" pod "f1ca4728-e019-4147-b9a3-d3f662d98e77" (UID: "f1ca4728-e019-4147-b9a3-d3f662d98e77"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:26.837640 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:26.837294    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1ca4728-e019-4147-b9a3-d3f662d98e77-istiod-ca-cert" pod "f1ca4728-e019-4147-b9a3-d3f662d98e77" (UID: "f1ca4728-e019-4147-b9a3-d3f662d98e77"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:26.838724 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:26.838703    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1ca4728-e019-4147-b9a3-d3f662d98e77-istio-token" pod "f1ca4728-e019-4147-b9a3-d3f662d98e77" (UID: "f1ca4728-e019-4147-b9a3-d3f662d98e77"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:26.838905 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:26.838885    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1ca4728-e019-4147-b9a3-d3f662d98e77-kube-api-access-92nbc" pod "f1ca4728-e019-4147-b9a3-d3f662d98e77" (UID: "f1ca4728-e019-4147-b9a3-d3f662d98e77"). InnerVolumeSpecName "kube-api-access-92nbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:26.839305 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:26.839288    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1ca4728-e019-4147-b9a3-d3f662d98e77-istio-envoy" pod "f1ca4728-e019-4147-b9a3-d3f662d98e77" (UID: "f1ca4728-e019-4147-b9a3-d3f662d98e77"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:26.839695 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:26.839682    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1ca4728-e019-4147-b9a3-d3f662d98e77-istio-podinfo" pod "f1ca4728-e019-4147-b9a3-d3f662d98e77" (UID: "f1ca4728-e019-4147-b9a3-d3f662d98e77"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:46:26.937750 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:26.937707    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1ca4728-e019-4147-b9a3-d3f662d98e77-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:26.937750 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:26.937754    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92nbc\" (UniqueName: \"kubernetes.io/projected/f1ca4728-e019-4147-b9a3-d3f662d98e77-kube-api-access-92nbc\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:26.937968 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:26.937766    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1ca4728-e019-4147-b9a3-d3f662d98e77-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:26.937968 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:26.937777    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ca4728-e019-4147-b9a3-d3f662d98e77-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:26.937968 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:26.937786    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ca4728-e019-4147-b9a3-d3f662d98e77-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:26.937968 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:26.937794    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1ca4728-e019-4147-b9a3-d3f662d98e77-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:26.937968 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:26.937802    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1ca4728-e019-4147-b9a3-d3f662d98e77-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:27.196813 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:27.196080    5043 generic.go:356] "Generic (PLEG): container finished" podID="f1ca4728-e019-4147-b9a3-d3f662d98e77" containerID="6927cc37fe5ae78876e2ef49ba2720ba7947b71ead78bcdc4bb804a20073f013" exitCode=0
Oct 30 19:46:27.196813 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:27.196179    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx" event={"ID":"f1ca4728-e019-4147-b9a3-d3f662d98e77","Type":"ContainerDied","Data":"6927cc37fe5ae78876e2ef49ba2720ba7947b71ead78bcdc4bb804a20073f013"}
Oct 30 19:46:27.196813 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:27.196236    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx" event={"ID":"f1ca4728-e019-4147-b9a3-d3f662d98e77","Type":"ContainerDied","Data":"5c2965dadde5d1cf97a7999a92b1699c1c664594ce6358ced7b24a11b2a83af4"}
Oct 30 19:46:27.196813 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:27.196268    5043 scope.go:122] "RemoveContainer" containerID="6927cc37fe5ae78876e2ef49ba2720ba7947b71ead78bcdc4bb804a20073f013"
Oct 30 19:46:27.196813 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:27.196397    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx"
Oct 30 19:46:27.231687 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:27.231656    5043 scope.go:122] "RemoveContainer" containerID="e5e78c0bd3b1fa3bd878f6ac7958d9146cfea1dbaa9cdc918d7e310f9098d798"
Oct 30 19:46:27.231845 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:27.231775    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx"]
Oct 30 19:46:27.236655 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:27.236628    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-4g5wx"]
Oct 30 19:46:27.244584 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:27.244561    5043 scope.go:122] "RemoveContainer" containerID="6927cc37fe5ae78876e2ef49ba2720ba7947b71ead78bcdc4bb804a20073f013"
Oct 30 19:46:27.244934 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:46:27.244910    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6927cc37fe5ae78876e2ef49ba2720ba7947b71ead78bcdc4bb804a20073f013\": container with ID starting with 6927cc37fe5ae78876e2ef49ba2720ba7947b71ead78bcdc4bb804a20073f013 not found: ID does not exist" containerID="6927cc37fe5ae78876e2ef49ba2720ba7947b71ead78bcdc4bb804a20073f013"
Oct 30 19:46:27.245015 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:27.244944    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6927cc37fe5ae78876e2ef49ba2720ba7947b71ead78bcdc4bb804a20073f013"} err="failed to get container status \"6927cc37fe5ae78876e2ef49ba2720ba7947b71ead78bcdc4bb804a20073f013\": rpc error: code = NotFound desc = could not find container \"6927cc37fe5ae78876e2ef49ba2720ba7947b71ead78bcdc4bb804a20073f013\": container with ID starting with 6927cc37fe5ae78876e2ef49ba2720ba7947b71ead78bcdc4bb804a20073f013 not found: ID does not exist"
Oct 30 19:46:27.245015 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:27.244964    5043 scope.go:122] "RemoveContainer" containerID="e5e78c0bd3b1fa3bd878f6ac7958d9146cfea1dbaa9cdc918d7e310f9098d798"
Oct 30 19:46:27.245255 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:46:27.245237    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5e78c0bd3b1fa3bd878f6ac7958d9146cfea1dbaa9cdc918d7e310f9098d798\": container with ID starting with e5e78c0bd3b1fa3bd878f6ac7958d9146cfea1dbaa9cdc918d7e310f9098d798 not found: ID does not exist" containerID="e5e78c0bd3b1fa3bd878f6ac7958d9146cfea1dbaa9cdc918d7e310f9098d798"
Oct 30 19:46:27.245301 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:27.245262    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5e78c0bd3b1fa3bd878f6ac7958d9146cfea1dbaa9cdc918d7e310f9098d798"} err="failed to get container status \"e5e78c0bd3b1fa3bd878f6ac7958d9146cfea1dbaa9cdc918d7e310f9098d798\": rpc error: code = NotFound desc = could not find container \"e5e78c0bd3b1fa3bd878f6ac7958d9146cfea1dbaa9cdc918d7e310f9098d798\": container with ID starting with e5e78c0bd3b1fa3bd878f6ac7958d9146cfea1dbaa9cdc918d7e310f9098d798 not found: ID does not exist"
Oct 30 19:46:28.644284 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:28.644250    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1ca4728-e019-4147-b9a3-d3f662d98e77" path="/var/lib/kubelet/pods/f1ca4728-e019-4147-b9a3-d3f662d98e77/volumes"
Oct 30 19:46:46.627406 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.627368    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492"]
Oct 30 19:46:46.627997 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.627972    5043 cpu_manager.go:414] "containerMap: removing container" podUID="c78899fb-4833-45cb-bb84-8c15ae19be7c" containerName="revoke-keys"
Oct 30 19:46:46.628174 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.628155    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c78899fb-4833-45cb-bb84-8c15ae19be7c" containerName="revoke-keys"
Oct 30 19:46:46.628174 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.628176    5043 cpu_manager.go:414] "containerMap: removing container" podUID="f1ca4728-e019-4147-b9a3-d3f662d98e77" containerName="istio-proxy"
Oct 30 19:46:46.628305 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.628183    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1ca4728-e019-4147-b9a3-d3f662d98e77" containerName="istio-proxy"
Oct 30 19:46:46.628305 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.628204    5043 cpu_manager.go:414] "containerMap: removing container" podUID="83c175fe-cdfd-4a54-88b6-f9a2061a11dd" containerName="payload-pre-processing"
Oct 30 19:46:46.628305 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.628211    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83c175fe-cdfd-4a54-88b6-f9a2061a11dd" containerName="payload-pre-processing"
Oct 30 19:46:46.628305 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.628224    5043 cpu_manager.go:414] "containerMap: removing container" podUID="f1ca4728-e019-4147-b9a3-d3f662d98e77" containerName="istio-proxy"
Oct 30 19:46:46.628305 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.628229    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1ca4728-e019-4147-b9a3-d3f662d98e77" containerName="istio-proxy"
Oct 30 19:46:46.628305 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.628240    5043 cpu_manager.go:414] "containerMap: removing container" podUID="33ae4933-1096-419e-81c6-defab7a8d6e0" containerName="payload-pre-processing"
Oct 30 19:46:46.628305 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.628244    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33ae4933-1096-419e-81c6-defab7a8d6e0" containerName="payload-pre-processing"
Oct 30 19:46:46.628305 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.628254    5043 cpu_manager.go:414] "containerMap: removing container" podUID="63688e51-3c1c-41ed-af40-4e2aa0e98c9e" containerName="istio-proxy"
Oct 30 19:46:46.628305 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.628261    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63688e51-3c1c-41ed-af40-4e2aa0e98c9e" containerName="istio-proxy"
Oct 30 19:46:46.628305 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.628284    5043 cpu_manager.go:414] "containerMap: removing container" podUID="5aef173f-b001-4494-89df-65d747c56b31" containerName="revoke-keys"
Oct 30 19:46:46.628305 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.628289    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aef173f-b001-4494-89df-65d747c56b31" containerName="revoke-keys"
Oct 30 19:46:46.628654 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.628381    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="83c175fe-cdfd-4a54-88b6-f9a2061a11dd" containerName="payload-pre-processing"
Oct 30 19:46:46.628654 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.628391    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1ca4728-e019-4147-b9a3-d3f662d98e77" containerName="istio-proxy"
Oct 30 19:46:46.628654 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.628397    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="33ae4933-1096-419e-81c6-defab7a8d6e0" containerName="payload-pre-processing"
Oct 30 19:46:46.628654 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.628403    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="63688e51-3c1c-41ed-af40-4e2aa0e98c9e" containerName="istio-proxy"
Oct 30 19:46:46.628654 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.628413    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aef173f-b001-4494-89df-65d747c56b31" containerName="revoke-keys"
Oct 30 19:46:46.628654 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.628420    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="c78899fb-4833-45cb-bb84-8c15ae19be7c" containerName="revoke-keys"
Oct 30 19:46:46.632034 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.632013    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492"
Oct 30 19:46:46.650057 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.650021    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492"]
Oct 30 19:46:46.736870 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.736830    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba6f075a-2d9f-43f3-900f-2eced4621260-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4492\" (UID: \"ba6f075a-2d9f-43f3-900f-2eced4621260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492"
Oct 30 19:46:46.737059 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.736876    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzfbk\" (UniqueName: \"kubernetes.io/projected/ba6f075a-2d9f-43f3-900f-2eced4621260-kube-api-access-mzfbk\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4492\" (UID: \"ba6f075a-2d9f-43f3-900f-2eced4621260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492"
Oct 30 19:46:46.737059 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.737000    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba6f075a-2d9f-43f3-900f-2eced4621260-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4492\" (UID: \"ba6f075a-2d9f-43f3-900f-2eced4621260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492"
Oct 30 19:46:46.737059 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.737049    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6f075a-2d9f-43f3-900f-2eced4621260-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4492\" (UID: \"ba6f075a-2d9f-43f3-900f-2eced4621260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492"
Oct 30 19:46:46.737221 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.737116    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6f075a-2d9f-43f3-900f-2eced4621260-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4492\" (UID: \"ba6f075a-2d9f-43f3-900f-2eced4621260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492"
Oct 30 19:46:46.737221 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.737192    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba6f075a-2d9f-43f3-900f-2eced4621260-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4492\" (UID: \"ba6f075a-2d9f-43f3-900f-2eced4621260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492"
Oct 30 19:46:46.737311 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.737231    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba6f075a-2d9f-43f3-900f-2eced4621260-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4492\" (UID: \"ba6f075a-2d9f-43f3-900f-2eced4621260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492"
Oct 30 19:46:46.737311 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.737253    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba6f075a-2d9f-43f3-900f-2eced4621260-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4492\" (UID: \"ba6f075a-2d9f-43f3-900f-2eced4621260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492"
Oct 30 19:46:46.737392 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.737343    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba6f075a-2d9f-43f3-900f-2eced4621260-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4492\" (UID: \"ba6f075a-2d9f-43f3-900f-2eced4621260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492"
Oct 30 19:46:46.838764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.838717    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba6f075a-2d9f-43f3-900f-2eced4621260-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4492\" (UID: \"ba6f075a-2d9f-43f3-900f-2eced4621260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492"
Oct 30 19:46:46.838764 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.838771    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba6f075a-2d9f-43f3-900f-2eced4621260-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4492\" (UID: \"ba6f075a-2d9f-43f3-900f-2eced4621260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492"
Oct 30 19:46:46.839054 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.838806    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba6f075a-2d9f-43f3-900f-2eced4621260-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4492\" (UID: \"ba6f075a-2d9f-43f3-900f-2eced4621260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492"
Oct 30 19:46:46.839054 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.838842    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba6f075a-2d9f-43f3-900f-2eced4621260-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4492\" (UID: \"ba6f075a-2d9f-43f3-900f-2eced4621260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492"
Oct 30 19:46:46.839054 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.838860    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzfbk\" (UniqueName: \"kubernetes.io/projected/ba6f075a-2d9f-43f3-900f-2eced4621260-kube-api-access-mzfbk\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4492\" (UID: \"ba6f075a-2d9f-43f3-900f-2eced4621260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492"
Oct 30 19:46:46.839054 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.838934    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba6f075a-2d9f-43f3-900f-2eced4621260-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4492\" (UID: \"ba6f075a-2d9f-43f3-900f-2eced4621260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492"
Oct 30 19:46:46.839054 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.838980    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6f075a-2d9f-43f3-900f-2eced4621260-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4492\" (UID: \"ba6f075a-2d9f-43f3-900f-2eced4621260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492"
Oct 30 19:46:46.839054 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.839023    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6f075a-2d9f-43f3-900f-2eced4621260-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4492\" (UID: \"ba6f075a-2d9f-43f3-900f-2eced4621260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492"
Oct 30 19:46:46.839054 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.839052    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba6f075a-2d9f-43f3-900f-2eced4621260-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4492\" (UID: \"ba6f075a-2d9f-43f3-900f-2eced4621260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492"
Oct 30 19:46:46.839447 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.839391    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba6f075a-2d9f-43f3-900f-2eced4621260-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4492\" (UID: \"ba6f075a-2d9f-43f3-900f-2eced4621260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492"
Oct 30 19:46:46.839447 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.839434    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6f075a-2d9f-43f3-900f-2eced4621260-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4492\" (UID: \"ba6f075a-2d9f-43f3-900f-2eced4621260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492"
Oct 30 19:46:46.839631 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.839610    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba6f075a-2d9f-43f3-900f-2eced4621260-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4492\" (UID: \"ba6f075a-2d9f-43f3-900f-2eced4621260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492"
Oct 30 19:46:46.839702 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.839648    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6f075a-2d9f-43f3-900f-2eced4621260-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4492\" (UID: \"ba6f075a-2d9f-43f3-900f-2eced4621260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492"
Oct 30 19:46:46.839702 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.839670    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba6f075a-2d9f-43f3-900f-2eced4621260-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4492\" (UID: \"ba6f075a-2d9f-43f3-900f-2eced4621260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492"
Oct 30 19:46:46.842092 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.842070    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba6f075a-2d9f-43f3-900f-2eced4621260-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4492\" (UID: \"ba6f075a-2d9f-43f3-900f-2eced4621260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492"
Oct 30 19:46:46.842407 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.842385    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba6f075a-2d9f-43f3-900f-2eced4621260-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4492\" (UID: \"ba6f075a-2d9f-43f3-900f-2eced4621260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492"
Oct 30 19:46:46.847890 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.847868    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba6f075a-2d9f-43f3-900f-2eced4621260-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4492\" (UID: \"ba6f075a-2d9f-43f3-900f-2eced4621260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492"
Oct 30 19:46:46.848572 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.848552    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzfbk\" (UniqueName: \"kubernetes.io/projected/ba6f075a-2d9f-43f3-900f-2eced4621260-kube-api-access-mzfbk\") pod \"maas-default-gateway-openshift-default-5bc884dd87-q4492\" (UID: \"ba6f075a-2d9f-43f3-900f-2eced4621260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492"
Oct 30 19:46:46.946962 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:46.946876    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492"
Oct 30 19:46:47.251425 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:47.251397    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492"]
Oct 30 19:46:47.253766 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:46:47.253712    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba6f075a_2d9f_43f3_900f_2eced4621260.slice/crio-bf294ee1b644e17facbf07bc4fe165a7ddee7a9daf1985176fb2578574d07bcb WatchSource:0}: Error finding container bf294ee1b644e17facbf07bc4fe165a7ddee7a9daf1985176fb2578574d07bcb: Status 404 returned error can't find the container with id bf294ee1b644e17facbf07bc4fe165a7ddee7a9daf1985176fb2578574d07bcb
Oct 30 19:46:47.256043 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:47.256009    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:46:47.256187 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:47.256074    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:46:47.256187 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:47.256104    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:46:47.339099 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:47.339057    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492" event={"ID":"ba6f075a-2d9f-43f3-900f-2eced4621260","Type":"ContainerStarted","Data":"67660a99c222247c42afaa11f8c8bf203d9ca9272f7565c7bb01c5a46a02c599"}
Oct 30 19:46:47.339300 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:47.339157    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492" event={"ID":"ba6f075a-2d9f-43f3-900f-2eced4621260","Type":"ContainerStarted","Data":"bf294ee1b644e17facbf07bc4fe165a7ddee7a9daf1985176fb2578574d07bcb"}
Oct 30 19:46:47.370688 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:47.370631    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492" podStartSLOduration=1.370617229 podStartE2EDuration="1.370617229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:47.368510738 +0000 UTC m=+1826.879570704" watchObservedRunningTime="2026-10-30 19:46:47.370617229 +0000 UTC m=+1826.881677193"
Oct 30 19:46:47.947997 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:47.947957    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492"
Oct 30 19:46:48.949621 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:48.949583    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-q4492 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body=
Oct 30 19:46:48.950117 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:48.949641    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492" podUID="ba6f075a-2d9f-43f3-900f-2eced4621260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused"
Oct 30 19:46:49.948147 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:49.948113    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-q4492 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body=
Oct 30 19:46:49.948343 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:49.948173    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492" podUID="ba6f075a-2d9f-43f3-900f-2eced4621260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused"
Oct 30 19:46:50.948097 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:50.948061    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-q4492 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body=
Oct 30 19:46:50.948500 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:50.948114    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492" podUID="ba6f075a-2d9f-43f3-900f-2eced4621260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused"
Oct 30 19:46:51.948142 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:51.948103    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-q4492 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body=
Oct 30 19:46:51.948551 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:51.948159    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492" podUID="ba6f075a-2d9f-43f3-900f-2eced4621260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused"
Oct 30 19:46:52.948131 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:52.948090    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-q4492 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body=
Oct 30 19:46:52.948402 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:52.948160    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492" podUID="ba6f075a-2d9f-43f3-900f-2eced4621260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused"
Oct 30 19:46:53.947804 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:53.947767    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-q4492 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body=
Oct 30 19:46:53.947995 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:53.947834    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492" podUID="ba6f075a-2d9f-43f3-900f-2eced4621260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused"
Oct 30 19:46:54.977219 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:46:54.977175    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:46:55.687322 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:55.687281    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492"
Oct 30 19:46:55.688131 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:55.688096    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492"
Oct 30 19:46:55.689225 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:46:55.689197    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492"
Oct 30 19:47:24.976927 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:47:24.976883    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:47:25.184328 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:47:25.184296    5043 scope.go:122] "RemoveContainer" containerID="846ca16a269d995d1cdfeae2a29aab3f6bc1901261c8681c2d2b68b09ce76cfe"
Oct 30 19:47:31.025148 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:47:31.025109    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbb1760d-v497b"]
Oct 30 19:47:31.028534 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:47:31.028503    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b489b261-cc58d"]
Oct 30 19:47:31.031278 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:47:31.031253    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b489b261-cc58d"]
Oct 30 19:47:31.033424 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:47:31.033403    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbb1760d-v497b"]
Oct 30 19:47:32.644379 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:47:32.644347    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="448520e8-011d-4079-b0b2-ddb5263fd382" path="/var/lib/kubelet/pods/448520e8-011d-4079-b0b2-ddb5263fd382/volumes"
Oct 30 19:47:32.644799 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:47:32.644768    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2d3d64b-2e22-491b-a612-078423f87212" path="/var/lib/kubelet/pods/a2d3d64b-2e22-491b-a612-078423f87212/volumes"
Oct 30 19:47:54.976227 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:47:54.976181    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:48:06.035618 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:48:06.035582    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab530c70-xwg6v"]
Oct 30 19:48:06.038814 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:48:06.038785    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab530c70-xwg6v"]
Oct 30 19:48:06.644001 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:48:06.643966    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6372f52-7e2d-4118-82f7-8b9a20e5c0fc" path="/var/lib/kubelet/pods/e6372f52-7e2d-4118-82f7-8b9a20e5c0fc/volumes"
Oct 30 19:48:13.285761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:48:13.284851    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-f5qx7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:48:13.285761 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:48:13.284935    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7" podUID="91978947-9c53-4a67-8f97-36ce17619c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:13.524013 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:48:13.523971    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qj2g6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:48:13.524368 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:48:13.524334    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6" podUID="9f2b4d7b-0014-49da-b5c2-0cf1944e8796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:17.025575 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:48:17.025540    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f06bdefe-8blkv"]
Oct 30 19:48:17.028303 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:48:17.028276    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f06bdefe-8blkv"]
Oct 30 19:48:18.643948 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:48:18.643913    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64f45468-8b8a-4191-a217-b7b6993d94ed" path="/var/lib/kubelet/pods/64f45468-8b8a-4191-a217-b7b6993d94ed/volumes"
Oct 30 19:48:24.977025 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:48:24.976990    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:48:25.213595 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:48:25.213564    5043 scope.go:122] "RemoveContainer" containerID="38b56a8722633fab4e8704f17df2dd481c263aa9888bb9585fbba92601b20ef1"
Oct 30 19:48:25.223502 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:48:25.223482    5043 scope.go:122] "RemoveContainer" containerID="dd63b14f79ec2e3d0e33e3fc7f1d05fea61742ecd679eddae0a83894802b14cc"
Oct 30 19:48:32.032983 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:48:32.032904    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41871c4d-4mpdm"]
Oct 30 19:48:32.035312 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:48:32.035290    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41871c4d-4mpdm"]
Oct 30 19:48:32.644237 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:48:32.644203    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5" path="/var/lib/kubelet/pods/aa0c5459-e25b-451b-ba4f-d2a6d9cd9ec5/volumes"
Oct 30 19:48:39.025054 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:48:39.025015    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1582090b-95fsw"]
Oct 30 19:48:39.027953 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:48:39.027930    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1582090b-95fsw"]
Oct 30 19:48:40.644366 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:48:40.644332    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74855f09-5dc1-4dea-81e4-8dd04e20d7c0" path="/var/lib/kubelet/pods/74855f09-5dc1-4dea-81e4-8dd04e20d7c0/volumes"
Oct 30 19:48:43.033878 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:48:43.033833    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a66b56a0-mrbtw"]
Oct 30 19:48:43.041340 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:48:43.041312    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a66b56a0-mrbtw"]
Oct 30 19:48:44.643873 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:48:44.643842    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11e9713c-21d8-495e-9ed4-a94017cb4f56" path="/var/lib/kubelet/pods/11e9713c-21d8-495e-9ed4-a94017cb4f56/volumes"
Oct 30 19:48:54.025781 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:48:54.025748    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8735990a-hnfjb"]
Oct 30 19:48:54.030091 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:48:54.030065    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8735990a-hnfjb"]
Oct 30 19:48:54.644134 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:48:54.644099    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bffdd089-54cb-40c3-8c7d-cc797874069b" path="/var/lib/kubelet/pods/bffdd089-54cb-40c3-8c7d-cc797874069b/volumes"
Oct 30 19:48:54.976566 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:48:54.976531    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:49:24.975581 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:49:24.975543    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:49:25.251490 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:49:25.251410    5043 scope.go:122] "RemoveContainer" containerID="8849943ec72f7ac027b43c6d7883e866bf9cbb2d6cacad33d734aa168cf7624d"
Oct 30 19:49:25.261347 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:49:25.261329    5043 scope.go:122] "RemoveContainer" containerID="61d83a1f308297eb84a8f91542319f30c10099fdbd43e18617e00df725e99527"
Oct 30 19:49:25.270390 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:49:25.270369    5043 scope.go:122] "RemoveContainer" containerID="08d60fe43e98fc3a5ed05f53713bf723a3c33a2ffd0133545201e9e72dcd2f2d"
Oct 30 19:49:25.278809 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:49:25.278792    5043 scope.go:122] "RemoveContainer" containerID="e42e23fb73a05f0c625b3dd00d9ffeca4fa0d9d52c2bb88d45a6b15eb0eb0206"
Oct 30 19:49:25.287219 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:49:25.287202    5043 scope.go:122] "RemoveContainer" containerID="3e6cc256fc8e08ee6ddfbc8f09156772efcd2374dc23a2107e8114f45b12b9b4"
Oct 30 19:49:40.064101 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:49:40.064063    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a97ffc3b-2lkdh"]
Oct 30 19:49:40.080661 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:49:40.080629    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a97ffc3b-2lkdh"]
Oct 30 19:49:40.652567 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:49:40.652224    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f4e49b7-9e2c-4479-a189-92a079a19e30" path="/var/lib/kubelet/pods/9f4e49b7-9e2c-4479-a189-92a079a19e30/volumes"
Oct 30 19:49:44.024065 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:49:44.024022    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a544ff33-2sfhh"]
Oct 30 19:49:44.027324 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:49:44.027297    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e142711-bpvzp"]
Oct 30 19:49:44.029795 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:49:44.029772    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a544ff33-2sfhh"]
Oct 30 19:49:44.031817 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:49:44.031793    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e142711-bpvzp"]
Oct 30 19:49:44.643708 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:49:44.643677    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="358f00f6-d477-4a66-917d-d551097c6bcc" path="/var/lib/kubelet/pods/358f00f6-d477-4a66-917d-d551097c6bcc/volumes"
Oct 30 19:49:44.644140 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:49:44.644126    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5279675-574c-4cee-9a73-b4b18429b99d" path="/var/lib/kubelet/pods/f5279675-574c-4cee-9a73-b4b18429b99d/volumes"
Oct 30 19:49:53.033002 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:49:53.032966    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaae805a-m4l2s"]
Oct 30 19:49:53.041051 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:49:53.041023    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaae805a-m4l2s"]
Oct 30 19:49:54.643946 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:49:54.643909    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14a645f0-5289-46a9-ab64-2743663bb5d0" path="/var/lib/kubelet/pods/14a645f0-5289-46a9-ab64-2743663bb5d0/volumes"
Oct 30 19:49:54.978188 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:49:54.978154    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:50:12.038475 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:50:12.038426    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaeb877c-szszg"]
Oct 30 19:50:12.040345 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:50:12.040321    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84924f45-h5mdf"]
Oct 30 19:50:12.045834 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:50:12.045799    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaeb877c-szszg"]
Oct 30 19:50:12.049005 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:50:12.048974    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84924f45-h5mdf"]
Oct 30 19:50:12.646626 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:50:12.646591    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4db4df1f-ce94-4cde-9e4e-b623f5ab1edd" path="/var/lib/kubelet/pods/4db4df1f-ce94-4cde-9e4e-b623f5ab1edd/volumes"
Oct 30 19:50:12.647205 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:50:12.647188    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afb97d96-82b4-4de1-b175-f02025aba9ad" path="/var/lib/kubelet/pods/afb97d96-82b4-4de1-b175-f02025aba9ad/volumes"
Oct 30 19:50:25.048799 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:50:25.048758    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:50:25.333813 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:50:25.329793    5043 scope.go:122] "RemoveContainer" containerID="1bf504fdbd6b15f9cc35713ca798432f9cc87c2226ba368fd3b6ed5ead8d5efd"
Oct 30 19:50:25.350836 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:50:25.350807    5043 scope.go:122] "RemoveContainer" containerID="8ba2e53999da8deb64dc6b9eeb950178d9298c6f9da9438878ed15c81c326e11"
Oct 30 19:50:25.365650 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:50:25.365588    5043 scope.go:122] "RemoveContainer" containerID="06f13c02498b60fc779f547f449d5caf499c83b45904191bd5db9798c0d3c7cd"
Oct 30 19:50:25.419012 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:50:25.418967    5043 scope.go:122] "RemoveContainer" containerID="b28b22756fe6e624347d6edde315cd5cad6caea331500b4a317d92da1891528d"
Oct 30 19:50:25.457417 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:50:25.457262    5043 scope.go:122] "RemoveContainer" containerID="fd4aa340d89f4d52f3f2dafd088d63d2eb016152599dab59b6bb5c4592a1c3ae"
Oct 30 19:50:25.499728 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:50:25.499587    5043 scope.go:122] "RemoveContainer" containerID="cceb9477ecf0e85c74e62bce8862c3dd04b6eb59d8fe30ce78fbb2ed05c37556"
Oct 30 19:50:25.544763 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:50:25.544702    5043 scope.go:122] "RemoveContainer" containerID="dae1ddb6be0aec4fc67a7f9f6a1a4e1fbc72c0308f6bb8dadcd881569e02a3b1"
Oct 30 19:50:25.554910 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:50:25.554881    5043 scope.go:122] "RemoveContainer" containerID="70dbbc48a0dc64c23ddc584d5984228af148062593220caed5e0fdbca84e3fc0"
Oct 30 19:50:28.287899 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:50:28.287840    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-f5qx7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:50:28.288544 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:50:28.288507    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7" podUID="91978947-9c53-4a67-8f97-36ce17619c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:28.529920 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:50:28.529879    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qj2g6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:50:28.530102 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:50:28.529949    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6" podUID="9f2b4d7b-0014-49da-b5c2-0cf1944e8796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:50:53.027779 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:50:53.027726    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1366788f-zl2qk"]
Oct 30 19:50:53.034134 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:50:53.034098    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1366788f-zl2qk"]
Oct 30 19:50:54.644355 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:50:54.644321    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fca56991-a8aa-48ff-b1ad-99443549fa8c" path="/var/lib/kubelet/pods/fca56991-a8aa-48ff-b1ad-99443549fa8c/volumes"
Oct 30 19:50:54.983969 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:50:54.983928    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:51:03.026847 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:51:03.026798    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-803dc14d-jqdgz"]
Oct 30 19:51:03.028243 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:51:03.028216    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-803dc14d-jqdgz"]
Oct 30 19:51:04.046146 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:51:04.046109    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5109a2f-kzgdn"]
Oct 30 19:51:04.061674 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:51:04.061641    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5109a2f-kzgdn"]
Oct 30 19:51:04.643621 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:51:04.643585    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5aef173f-b001-4494-89df-65d747c56b31" path="/var/lib/kubelet/pods/5aef173f-b001-4494-89df-65d747c56b31/volumes"
Oct 30 19:51:04.644054 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:51:04.644039    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c78899fb-4833-45cb-bb84-8c15ae19be7c" path="/var/lib/kubelet/pods/c78899fb-4833-45cb-bb84-8c15ae19be7c/volumes"
Oct 30 19:51:21.218113 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:51:21.218072    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cmw6w_76e21916-0744-4499-a0cb-64a7a44a8285/console-operator/2.log"
Oct 30 19:51:21.221245 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:51:21.221222    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cmw6w_76e21916-0744-4499-a0cb-64a7a44a8285/console-operator/2.log"
Oct 30 19:51:24.976610 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:51:24.976556    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:51:25.770187 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:51:25.770153    5043 scope.go:122] "RemoveContainer" containerID="755b146060b518b204a514a2e189a1e2dbc126a51290640ddedc3203e9c55e4c"
Oct 30 19:51:25.780052 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:51:25.780026    5043 scope.go:122] "RemoveContainer" containerID="2893837ec98cdd4f8ab3c641f230f1949393ac1fa1471c7ea2762326b7160b37"
Oct 30 19:51:25.789816 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:51:25.789799    5043 scope.go:122] "RemoveContainer" containerID="472ebf84359c01ca2d015a9fdcd914a1062e20dd8f483d6036989d602ebb9efd"
Oct 30 19:51:25.798980 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:51:25.798960    5043 scope.go:122] "RemoveContainer" containerID="5fa07d580300dc27006927ed3066c57b1751ece35b0bef03619e29697dc9ac02"
Oct 30 19:51:25.809488 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:51:25.809467    5043 scope.go:122] "RemoveContainer" containerID="f1c43b6ecb45bd3c2addb9ef0f16d606d5df54076c18fb58fb53dd06525d4b9a"
Oct 30 19:51:25.821510 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:51:25.819909    5043 scope.go:122] "RemoveContainer" containerID="6850414af39877df6865796cdeb6356dbefd3b497adc0b6408addef5bc461ac6"
Oct 30 19:51:25.829910 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:51:25.829885    5043 scope.go:122] "RemoveContainer" containerID="9c571999adeb6a26ef885dd09fc1048a7cf31d24b4e7351856332a2b48460a53"
Oct 30 19:51:25.838868 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:51:25.838845    5043 scope.go:122] "RemoveContainer" containerID="d7860678b4b1b3ad0650ebd96fd6145e8586533e2ad0fa417b4476b00834fc07"
Oct 30 19:51:25.847117 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:51:25.847088    5043 scope.go:122] "RemoveContainer" containerID="7c93c003c43c9f3ffd8983e52e6768d9f1532cc7edcd688c728e86b782725820"
Oct 30 19:51:25.854855 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:51:25.854836    5043 scope.go:122] "RemoveContainer" containerID="04af014abfc908eb0679cabfaa48f5d6e79936ce0015cf73c3b1ced273b65220"
Oct 30 19:51:25.862603 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:51:25.862582    5043 scope.go:122] "RemoveContainer" containerID="37c1f7179985033b12dd2f89ba657862ea6a27b3972be1986ecb9f7439477f5c"
Oct 30 19:51:25.870612 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:51:25.870591    5043 scope.go:122] "RemoveContainer" containerID="b955bee388a3cca7c53db7a69f2e0dc5629cf8eb25ac2bdf73a9357504b5e64b"
Oct 30 19:51:54.976149 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:51:54.976114    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:52:24.976143 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:52:24.976104    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:52:25.936100 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:52:25.936066    5043 scope.go:122] "RemoveContainer" containerID="3e07fd0d14ca89a2cffbc25631e0458a8adff022a03abbd4a90d979961e0b1b3"
Oct 30 19:52:25.945198 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:52:25.945177    5043 scope.go:122] "RemoveContainer" containerID="895e1a6c43d723545fac739cbd49ae8df23565c184bd7515a5abe314406e0b87"
Oct 30 19:52:25.954927 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:52:25.954906    5043 scope.go:122] "RemoveContainer" containerID="5491144ca6d2af6af53ed2fe79781586dec14f5ad981271d97b6001ff78c5920"
Oct 30 19:52:25.965192 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:52:25.965168    5043 scope.go:122] "RemoveContainer" containerID="71601132b4a59dd38806cf63339ac3449f4068b3a3cb15c62348e1497370bb64"
Oct 30 19:52:25.975605 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:52:25.975584    5043 scope.go:122] "RemoveContainer" containerID="e36957a61cbe6d280391d5deb6d1289bf83c419080e33092f13b4f303112114c"
Oct 30 19:52:25.983965 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:52:25.983934    5043 scope.go:122] "RemoveContainer" containerID="0877106a08d4de32c0887a0f3cc22740074c951749a108878c293cfa5226f969"
Oct 30 19:52:54.976221 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:52:54.976179    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:53:24.977155 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:53:24.977068    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:53:54.976196 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:53:54.976161    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:54:24.976784 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:54:24.976746    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:54:54.975408 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:54:54.975365    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:55:24.975636 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:55:24.975594    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:55:54.976413 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:55:54.976377    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:56:21.249901 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:21.249805    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cmw6w_76e21916-0744-4499-a0cb-64a7a44a8285/console-operator/2.log"
Oct 30 19:56:21.253308 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:21.253285    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cmw6w_76e21916-0744-4499-a0cb-64a7a44a8285/console-operator/2.log"
Oct 30 19:56:24.977438 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:56:24.977400    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:56:47.290687 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:47.290636    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6"]
Oct 30 19:56:47.291375 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:47.290941    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6" podUID="9f2b4d7b-0014-49da-b5c2-0cf1944e8796" containerName="istio-proxy" containerID="cri-o://556e01ffb2066c8435506a921b3660f26e1038f7e9e1ee98fc6bacf6b5401a62" gracePeriod=30
Oct 30 19:56:47.292915 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:47.292886    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492"]
Oct 30 19:56:47.293188 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:47.293102    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492" podUID="ba6f075a-2d9f-43f3-900f-2eced4621260" containerName="istio-proxy" containerID="cri-o://67660a99c222247c42afaa11f8c8bf203d9ca9272f7565c7bb01c5a46a02c599" gracePeriod=30
Oct 30 19:56:47.298776 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:47.298695    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7"]
Oct 30 19:56:47.299021 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:47.298997    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7" podUID="91978947-9c53-4a67-8f97-36ce17619c17" containerName="istio-proxy" containerID="cri-o://808bbc92a0a8928e0fc8262ecaf57243be29af3000a90c546307d58bf27306bd" gracePeriod=30
Oct 30 19:56:52.600414 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.600321    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6"
Oct 30 19:56:52.628785 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.628722    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492"
Oct 30 19:56:52.634321 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.634293    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7"
Oct 30 19:56:52.748089 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.748052    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91978947-9c53-4a67-8f97-36ce17619c17-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91978947-9c53-4a67-8f97-36ce17619c17-workload-socket\") pod \"91978947-9c53-4a67-8f97-36ce17619c17\" (UID: \"91978947-9c53-4a67-8f97-36ce17619c17\") "
Oct 30 19:56:52.748301 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.748105    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-workload-certs\") pod \"9f2b4d7b-0014-49da-b5c2-0cf1944e8796\" (UID: \"9f2b4d7b-0014-49da-b5c2-0cf1944e8796\") "
Oct 30 19:56:52.748301 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.748128    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba6f075a-2d9f-43f3-900f-2eced4621260-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba6f075a-2d9f-43f3-900f-2eced4621260-istio-envoy\") pod \"ba6f075a-2d9f-43f3-900f-2eced4621260\" (UID: \"ba6f075a-2d9f-43f3-900f-2eced4621260\") "
Oct 30 19:56:52.748301 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.748158    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91978947-9c53-4a67-8f97-36ce17619c17-kube-api-access-j29w2\" (UniqueName: \"kubernetes.io/projected/91978947-9c53-4a67-8f97-36ce17619c17-kube-api-access-j29w2\") pod \"91978947-9c53-4a67-8f97-36ce17619c17\" (UID: \"91978947-9c53-4a67-8f97-36ce17619c17\") "
Oct 30 19:56:52.748301 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.748178    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba6f075a-2d9f-43f3-900f-2eced4621260-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6f075a-2d9f-43f3-900f-2eced4621260-workload-socket\") pod \"ba6f075a-2d9f-43f3-900f-2eced4621260\" (UID: \"ba6f075a-2d9f-43f3-900f-2eced4621260\") "
Oct 30 19:56:52.748301 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.748197    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba6f075a-2d9f-43f3-900f-2eced4621260-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba6f075a-2d9f-43f3-900f-2eced4621260-workload-certs\") pod \"ba6f075a-2d9f-43f3-900f-2eced4621260\" (UID: \"ba6f075a-2d9f-43f3-900f-2eced4621260\") "
Oct 30 19:56:52.748301 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.748220    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba6f075a-2d9f-43f3-900f-2eced4621260-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba6f075a-2d9f-43f3-900f-2eced4621260-istio-data\") pod \"ba6f075a-2d9f-43f3-900f-2eced4621260\" (UID: \"ba6f075a-2d9f-43f3-900f-2eced4621260\") "
Oct 30 19:56:52.748301 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.748245    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91978947-9c53-4a67-8f97-36ce17619c17-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91978947-9c53-4a67-8f97-36ce17619c17-istio-data\") pod \"91978947-9c53-4a67-8f97-36ce17619c17\" (UID: \"91978947-9c53-4a67-8f97-36ce17619c17\") "
Oct 30 19:56:52.748753 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.748400    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91978947-9c53-4a67-8f97-36ce17619c17-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91978947-9c53-4a67-8f97-36ce17619c17-istio-podinfo\") pod \"91978947-9c53-4a67-8f97-36ce17619c17\" (UID: \"91978947-9c53-4a67-8f97-36ce17619c17\") "
Oct 30 19:56:52.748753 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.748550    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-istio-data\") pod \"9f2b4d7b-0014-49da-b5c2-0cf1944e8796\" (UID: \"9f2b4d7b-0014-49da-b5c2-0cf1944e8796\") "
Oct 30 19:56:52.748753 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.748408    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91978947-9c53-4a67-8f97-36ce17619c17-workload-socket" pod "91978947-9c53-4a67-8f97-36ce17619c17" (UID: "91978947-9c53-4a67-8f97-36ce17619c17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:52.748753 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.748616    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba6f075a-2d9f-43f3-900f-2eced4621260-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6f075a-2d9f-43f3-900f-2eced4621260-credential-socket\") pod \"ba6f075a-2d9f-43f3-900f-2eced4621260\" (UID: \"ba6f075a-2d9f-43f3-900f-2eced4621260\") "
Oct 30 19:56:52.748753 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.748442    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-workload-certs" pod "9f2b4d7b-0014-49da-b5c2-0cf1944e8796" (UID: "9f2b4d7b-0014-49da-b5c2-0cf1944e8796"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:52.748753 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.748457    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba6f075a-2d9f-43f3-900f-2eced4621260-workload-socket" pod "ba6f075a-2d9f-43f3-900f-2eced4621260" (UID: "ba6f075a-2d9f-43f3-900f-2eced4621260"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:52.748753 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.748619    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91978947-9c53-4a67-8f97-36ce17619c17-istio-data" pod "91978947-9c53-4a67-8f97-36ce17619c17" (UID: "91978947-9c53-4a67-8f97-36ce17619c17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:52.748753 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.748648    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba6f075a-2d9f-43f3-900f-2eced4621260-istio-data" pod "ba6f075a-2d9f-43f3-900f-2eced4621260" (UID: "ba6f075a-2d9f-43f3-900f-2eced4621260"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:52.748753 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.748664    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-istio-token\" (UniqueName: \"kubernetes.io/projected/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-istio-token\") pod \"9f2b4d7b-0014-49da-b5c2-0cf1944e8796\" (UID: \"9f2b4d7b-0014-49da-b5c2-0cf1944e8796\") "
Oct 30 19:56:52.749276 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.748792    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91978947-9c53-4a67-8f97-36ce17619c17-istio-token\" (UniqueName: \"kubernetes.io/projected/91978947-9c53-4a67-8f97-36ce17619c17-istio-token\") pod \"91978947-9c53-4a67-8f97-36ce17619c17\" (UID: \"91978947-9c53-4a67-8f97-36ce17619c17\") "
Oct 30 19:56:52.749276 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.748823    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba6f075a-2d9f-43f3-900f-2eced4621260-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba6f075a-2d9f-43f3-900f-2eced4621260-istio-podinfo\") pod \"ba6f075a-2d9f-43f3-900f-2eced4621260\" (UID: \"ba6f075a-2d9f-43f3-900f-2eced4621260\") "
Oct 30 19:56:52.749276 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.748856    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba6f075a-2d9f-43f3-900f-2eced4621260-workload-certs" pod "ba6f075a-2d9f-43f3-900f-2eced4621260" (UID: "ba6f075a-2d9f-43f3-900f-2eced4621260"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:52.749276 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.748884    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba6f075a-2d9f-43f3-900f-2eced4621260-credential-socket" pod "ba6f075a-2d9f-43f3-900f-2eced4621260" (UID: "ba6f075a-2d9f-43f3-900f-2eced4621260"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:52.749276 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.748895    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba6f075a-2d9f-43f3-900f-2eced4621260-istio-token\" (UniqueName: \"kubernetes.io/projected/ba6f075a-2d9f-43f3-900f-2eced4621260-istio-token\") pod \"ba6f075a-2d9f-43f3-900f-2eced4621260\" (UID: \"ba6f075a-2d9f-43f3-900f-2eced4621260\") "
Oct 30 19:56:52.749276 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.748927    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91978947-9c53-4a67-8f97-36ce17619c17-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91978947-9c53-4a67-8f97-36ce17619c17-credential-socket\") pod \"91978947-9c53-4a67-8f97-36ce17619c17\" (UID: \"91978947-9c53-4a67-8f97-36ce17619c17\") "
Oct 30 19:56:52.749276 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.748968    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba6f075a-2d9f-43f3-900f-2eced4621260-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba6f075a-2d9f-43f3-900f-2eced4621260-istiod-ca-cert\") pod \"ba6f075a-2d9f-43f3-900f-2eced4621260\" (UID: \"ba6f075a-2d9f-43f3-900f-2eced4621260\") "
Oct 30 19:56:52.749276 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.749004    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91978947-9c53-4a67-8f97-36ce17619c17-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91978947-9c53-4a67-8f97-36ce17619c17-workload-certs\") pod \"91978947-9c53-4a67-8f97-36ce17619c17\" (UID: \"91978947-9c53-4a67-8f97-36ce17619c17\") "
Oct 30 19:56:52.749276 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.749035    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91978947-9c53-4a67-8f97-36ce17619c17-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91978947-9c53-4a67-8f97-36ce17619c17-istiod-ca-cert\") pod \"91978947-9c53-4a67-8f97-36ce17619c17\" (UID: \"91978947-9c53-4a67-8f97-36ce17619c17\") "
Oct 30 19:56:52.749276 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.749073    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-istiod-ca-cert\") pod \"9f2b4d7b-0014-49da-b5c2-0cf1944e8796\" (UID: \"9f2b4d7b-0014-49da-b5c2-0cf1944e8796\") "
Oct 30 19:56:52.749276 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.749083    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-istio-data" pod "9f2b4d7b-0014-49da-b5c2-0cf1944e8796" (UID: "9f2b4d7b-0014-49da-b5c2-0cf1944e8796"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:52.749276 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.749104    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-istio-envoy\") pod \"9f2b4d7b-0014-49da-b5c2-0cf1944e8796\" (UID: \"9f2b4d7b-0014-49da-b5c2-0cf1944e8796\") "
Oct 30 19:56:52.749276 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.749145    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91978947-9c53-4a67-8f97-36ce17619c17-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91978947-9c53-4a67-8f97-36ce17619c17-istio-envoy\") pod \"91978947-9c53-4a67-8f97-36ce17619c17\" (UID: \"91978947-9c53-4a67-8f97-36ce17619c17\") "
Oct 30 19:56:52.750209 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.749260    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-kube-api-access-xvsn4\" (UniqueName: \"kubernetes.io/projected/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-kube-api-access-xvsn4\") pod \"9f2b4d7b-0014-49da-b5c2-0cf1944e8796\" (UID: \"9f2b4d7b-0014-49da-b5c2-0cf1944e8796\") "
Oct 30 19:56:52.750209 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.749394    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba6f075a-2d9f-43f3-900f-2eced4621260-kube-api-access-mzfbk\" (UniqueName: \"kubernetes.io/projected/ba6f075a-2d9f-43f3-900f-2eced4621260-kube-api-access-mzfbk\") pod \"ba6f075a-2d9f-43f3-900f-2eced4621260\" (UID: \"ba6f075a-2d9f-43f3-900f-2eced4621260\") "
Oct 30 19:56:52.750209 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.749456    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-istio-podinfo\") pod \"9f2b4d7b-0014-49da-b5c2-0cf1944e8796\" (UID: \"9f2b4d7b-0014-49da-b5c2-0cf1944e8796\") "
Oct 30 19:56:52.750209 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.749495    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-credential-socket\") pod \"9f2b4d7b-0014-49da-b5c2-0cf1944e8796\" (UID: \"9f2b4d7b-0014-49da-b5c2-0cf1944e8796\") "
Oct 30 19:56:52.750209 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.749524    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-workload-socket\") pod \"9f2b4d7b-0014-49da-b5c2-0cf1944e8796\" (UID: \"9f2b4d7b-0014-49da-b5c2-0cf1944e8796\") "
Oct 30 19:56:52.750209 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.749623    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba6f075a-2d9f-43f3-900f-2eced4621260-istiod-ca-cert" pod "ba6f075a-2d9f-43f3-900f-2eced4621260" (UID: "ba6f075a-2d9f-43f3-900f-2eced4621260"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:56:52.750209 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.749885    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-workload-socket" pod "9f2b4d7b-0014-49da-b5c2-0cf1944e8796" (UID: "9f2b4d7b-0014-49da-b5c2-0cf1944e8796"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:52.750209 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.750066    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91978947-9c53-4a67-8f97-36ce17619c17-workload-certs" pod "91978947-9c53-4a67-8f97-36ce17619c17" (UID: "91978947-9c53-4a67-8f97-36ce17619c17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:52.750671 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.750238    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91978947-9c53-4a67-8f97-36ce17619c17-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:52.750671 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.750261    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:52.750671 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.750276    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91978947-9c53-4a67-8f97-36ce17619c17-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:52.750671 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.750290    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:52.750671 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.750305    5043 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6f075a-2d9f-43f3-900f-2eced4621260-workload-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:52.750671 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.750318    5043 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba6f075a-2d9f-43f3-900f-2eced4621260-workload-certs\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:52.750671 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.750331    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba6f075a-2d9f-43f3-900f-2eced4621260-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:52.750671 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.750345    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91978947-9c53-4a67-8f97-36ce17619c17-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:52.750671 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.750359    5043 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-istio-data\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:52.750671 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.750361    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91978947-9c53-4a67-8f97-36ce17619c17-istiod-ca-cert" pod "91978947-9c53-4a67-8f97-36ce17619c17" (UID: "91978947-9c53-4a67-8f97-36ce17619c17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:56:52.750671 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.750373    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6f075a-2d9f-43f3-900f-2eced4621260-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:52.750671 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.750387    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba6f075a-2d9f-43f3-900f-2eced4621260-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:52.751588 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.750886    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-istiod-ca-cert" pod "9f2b4d7b-0014-49da-b5c2-0cf1944e8796" (UID: "9f2b4d7b-0014-49da-b5c2-0cf1944e8796"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:56:52.753347 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.751802    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91978947-9c53-4a67-8f97-36ce17619c17-credential-socket" pod "91978947-9c53-4a67-8f97-36ce17619c17" (UID: "91978947-9c53-4a67-8f97-36ce17619c17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:52.753347 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.752533    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91978947-9c53-4a67-8f97-36ce17619c17-istio-podinfo" pod "91978947-9c53-4a67-8f97-36ce17619c17" (UID: "91978947-9c53-4a67-8f97-36ce17619c17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:56:52.754056 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.753890    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-credential-socket" pod "9f2b4d7b-0014-49da-b5c2-0cf1944e8796" (UID: "9f2b4d7b-0014-49da-b5c2-0cf1944e8796"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:52.754524 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.754342    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91978947-9c53-4a67-8f97-36ce17619c17-kube-api-access-j29w2" pod "91978947-9c53-4a67-8f97-36ce17619c17" (UID: "91978947-9c53-4a67-8f97-36ce17619c17"). InnerVolumeSpecName "kube-api-access-j29w2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:52.754524 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.754410    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba6f075a-2d9f-43f3-900f-2eced4621260-istio-token" pod "ba6f075a-2d9f-43f3-900f-2eced4621260" (UID: "ba6f075a-2d9f-43f3-900f-2eced4621260"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:52.754524 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.754429    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-istio-token" pod "9f2b4d7b-0014-49da-b5c2-0cf1944e8796" (UID: "9f2b4d7b-0014-49da-b5c2-0cf1944e8796"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:52.754524 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.754496    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba6f075a-2d9f-43f3-900f-2eced4621260-istio-podinfo" pod "ba6f075a-2d9f-43f3-900f-2eced4621260" (UID: "ba6f075a-2d9f-43f3-900f-2eced4621260"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:56:52.754910 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.754655    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba6f075a-2d9f-43f3-900f-2eced4621260-kube-api-access-mzfbk" pod "ba6f075a-2d9f-43f3-900f-2eced4621260" (UID: "ba6f075a-2d9f-43f3-900f-2eced4621260"). InnerVolumeSpecName "kube-api-access-mzfbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:52.754910 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.754772    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91978947-9c53-4a67-8f97-36ce17619c17-istio-token" pod "91978947-9c53-4a67-8f97-36ce17619c17" (UID: "91978947-9c53-4a67-8f97-36ce17619c17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:52.754910 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.754874    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba6f075a-2d9f-43f3-900f-2eced4621260-istio-envoy" pod "ba6f075a-2d9f-43f3-900f-2eced4621260" (UID: "ba6f075a-2d9f-43f3-900f-2eced4621260"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:52.755086 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.755068    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-istio-envoy" pod "9f2b4d7b-0014-49da-b5c2-0cf1944e8796" (UID: "9f2b4d7b-0014-49da-b5c2-0cf1944e8796"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:52.755320 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.755290    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-istio-podinfo" pod "9f2b4d7b-0014-49da-b5c2-0cf1944e8796" (UID: "9f2b4d7b-0014-49da-b5c2-0cf1944e8796"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:56:52.755915 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.755892    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-kube-api-access-xvsn4" pod "9f2b4d7b-0014-49da-b5c2-0cf1944e8796" (UID: "9f2b4d7b-0014-49da-b5c2-0cf1944e8796"). InnerVolumeSpecName "kube-api-access-xvsn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:52.756064 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.756047    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91978947-9c53-4a67-8f97-36ce17619c17-istio-envoy" pod "91978947-9c53-4a67-8f97-36ce17619c17" (UID: "91978947-9c53-4a67-8f97-36ce17619c17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:52.851681 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.851606    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba6f075a-2d9f-43f3-900f-2eced4621260-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:52.851681 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.851632    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j29w2\" (UniqueName: \"kubernetes.io/projected/91978947-9c53-4a67-8f97-36ce17619c17-kube-api-access-j29w2\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:52.851681 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.851643    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91978947-9c53-4a67-8f97-36ce17619c17-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:52.851681 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.851652    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:52.851681 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.851661    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91978947-9c53-4a67-8f97-36ce17619c17-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:52.851681 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.851669    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba6f075a-2d9f-43f3-900f-2eced4621260-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:52.851681 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.851677    5043 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba6f075a-2d9f-43f3-900f-2eced4621260-istio-token\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:52.851681 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.851685    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91978947-9c53-4a67-8f97-36ce17619c17-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:52.852119 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.851694    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91978947-9c53-4a67-8f97-36ce17619c17-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:52.852119 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.851701    5043 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-istiod-ca-cert\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:52.852119 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.851711    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:52.852119 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.851720    5043 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91978947-9c53-4a67-8f97-36ce17619c17-istio-envoy\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:52.852119 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.851728    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvsn4\" (UniqueName: \"kubernetes.io/projected/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-kube-api-access-xvsn4\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:52.852119 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.851756    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzfbk\" (UniqueName: \"kubernetes.io/projected/ba6f075a-2d9f-43f3-900f-2eced4621260-kube-api-access-mzfbk\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:52.852119 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.851766    5043 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-istio-podinfo\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:52.852119 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:52.851774    5043 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f2b4d7b-0014-49da-b5c2-0cf1944e8796-credential-socket\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:53.265158 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:53.265117    5043 generic.go:356] "Generic (PLEG): container finished" podID="91978947-9c53-4a67-8f97-36ce17619c17" containerID="808bbc92a0a8928e0fc8262ecaf57243be29af3000a90c546307d58bf27306bd" exitCode=0
Oct 30 19:56:53.265158 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:53.265164    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7"
Oct 30 19:56:53.265465 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:53.265212    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7" event={"ID":"91978947-9c53-4a67-8f97-36ce17619c17","Type":"ContainerDied","Data":"808bbc92a0a8928e0fc8262ecaf57243be29af3000a90c546307d58bf27306bd"}
Oct 30 19:56:53.265465 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:53.265280    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7" event={"ID":"91978947-9c53-4a67-8f97-36ce17619c17","Type":"ContainerDied","Data":"4e9f4945147d160af7faddb12c0dfcb61721070eedba23acb6ffa6b27fa954c1"}
Oct 30 19:56:53.265465 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:53.265313    5043 scope.go:122] "RemoveContainer" containerID="808bbc92a0a8928e0fc8262ecaf57243be29af3000a90c546307d58bf27306bd"
Oct 30 19:56:53.266959 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:53.266923    5043 generic.go:356] "Generic (PLEG): container finished" podID="ba6f075a-2d9f-43f3-900f-2eced4621260" containerID="67660a99c222247c42afaa11f8c8bf203d9ca9272f7565c7bb01c5a46a02c599" exitCode=0
Oct 30 19:56:53.267142 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:53.266990    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492"
Oct 30 19:56:53.267142 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:53.267042    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492" event={"ID":"ba6f075a-2d9f-43f3-900f-2eced4621260","Type":"ContainerDied","Data":"67660a99c222247c42afaa11f8c8bf203d9ca9272f7565c7bb01c5a46a02c599"}
Oct 30 19:56:53.267142 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:53.267088    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492" event={"ID":"ba6f075a-2d9f-43f3-900f-2eced4621260","Type":"ContainerDied","Data":"bf294ee1b644e17facbf07bc4fe165a7ddee7a9daf1985176fb2578574d07bcb"}
Oct 30 19:56:53.268541 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:53.268515    5043 generic.go:356] "Generic (PLEG): container finished" podID="9f2b4d7b-0014-49da-b5c2-0cf1944e8796" containerID="556e01ffb2066c8435506a921b3660f26e1038f7e9e1ee98fc6bacf6b5401a62" exitCode=0
Oct 30 19:56:53.268674 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:53.268605    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6" event={"ID":"9f2b4d7b-0014-49da-b5c2-0cf1944e8796","Type":"ContainerDied","Data":"556e01ffb2066c8435506a921b3660f26e1038f7e9e1ee98fc6bacf6b5401a62"}
Oct 30 19:56:53.268674 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:53.268646    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6" event={"ID":"9f2b4d7b-0014-49da-b5c2-0cf1944e8796","Type":"ContainerDied","Data":"fef565663acbbd414f361b5dde6c4432c087796cef2b19f4faface54bed7fc63"}
Oct 30 19:56:53.268841 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:53.268714    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6"
Oct 30 19:56:53.278923 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:53.278905    5043 scope.go:122] "RemoveContainer" containerID="808bbc92a0a8928e0fc8262ecaf57243be29af3000a90c546307d58bf27306bd"
Oct 30 19:56:53.279274 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:56:53.279248    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"808bbc92a0a8928e0fc8262ecaf57243be29af3000a90c546307d58bf27306bd\": container with ID starting with 808bbc92a0a8928e0fc8262ecaf57243be29af3000a90c546307d58bf27306bd not found: ID does not exist" containerID="808bbc92a0a8928e0fc8262ecaf57243be29af3000a90c546307d58bf27306bd"
Oct 30 19:56:53.279379 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:53.279288    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"808bbc92a0a8928e0fc8262ecaf57243be29af3000a90c546307d58bf27306bd"} err="failed to get container status \"808bbc92a0a8928e0fc8262ecaf57243be29af3000a90c546307d58bf27306bd\": rpc error: code = NotFound desc = could not find container \"808bbc92a0a8928e0fc8262ecaf57243be29af3000a90c546307d58bf27306bd\": container with ID starting with 808bbc92a0a8928e0fc8262ecaf57243be29af3000a90c546307d58bf27306bd not found: ID does not exist"
Oct 30 19:56:53.279379 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:53.279312    5043 scope.go:122] "RemoveContainer" containerID="67660a99c222247c42afaa11f8c8bf203d9ca9272f7565c7bb01c5a46a02c599"
Oct 30 19:56:53.288986 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:53.288945    5043 scope.go:122] "RemoveContainer" containerID="67660a99c222247c42afaa11f8c8bf203d9ca9272f7565c7bb01c5a46a02c599"
Oct 30 19:56:53.289298 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:56:53.289275    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67660a99c222247c42afaa11f8c8bf203d9ca9272f7565c7bb01c5a46a02c599\": container with ID starting with 67660a99c222247c42afaa11f8c8bf203d9ca9272f7565c7bb01c5a46a02c599 not found: ID does not exist" containerID="67660a99c222247c42afaa11f8c8bf203d9ca9272f7565c7bb01c5a46a02c599"
Oct 30 19:56:53.289354 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:53.289310    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67660a99c222247c42afaa11f8c8bf203d9ca9272f7565c7bb01c5a46a02c599"} err="failed to get container status \"67660a99c222247c42afaa11f8c8bf203d9ca9272f7565c7bb01c5a46a02c599\": rpc error: code = NotFound desc = could not find container \"67660a99c222247c42afaa11f8c8bf203d9ca9272f7565c7bb01c5a46a02c599\": container with ID starting with 67660a99c222247c42afaa11f8c8bf203d9ca9272f7565c7bb01c5a46a02c599 not found: ID does not exist"
Oct 30 19:56:53.289354 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:53.289332    5043 scope.go:122] "RemoveContainer" containerID="556e01ffb2066c8435506a921b3660f26e1038f7e9e1ee98fc6bacf6b5401a62"
Oct 30 19:56:53.301879 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:53.298660    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492"]
Oct 30 19:56:53.301879 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:53.301551    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-q4492"]
Oct 30 19:56:53.302788 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:53.302769    5043 scope.go:122] "RemoveContainer" containerID="556e01ffb2066c8435506a921b3660f26e1038f7e9e1ee98fc6bacf6b5401a62"
Oct 30 19:56:53.303144 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:56:53.303120    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"556e01ffb2066c8435506a921b3660f26e1038f7e9e1ee98fc6bacf6b5401a62\": container with ID starting with 556e01ffb2066c8435506a921b3660f26e1038f7e9e1ee98fc6bacf6b5401a62 not found: ID does not exist" containerID="556e01ffb2066c8435506a921b3660f26e1038f7e9e1ee98fc6bacf6b5401a62"
Oct 30 19:56:53.303191 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:53.303153    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"556e01ffb2066c8435506a921b3660f26e1038f7e9e1ee98fc6bacf6b5401a62"} err="failed to get container status \"556e01ffb2066c8435506a921b3660f26e1038f7e9e1ee98fc6bacf6b5401a62\": rpc error: code = NotFound desc = could not find container \"556e01ffb2066c8435506a921b3660f26e1038f7e9e1ee98fc6bacf6b5401a62\": container with ID starting with 556e01ffb2066c8435506a921b3660f26e1038f7e9e1ee98fc6bacf6b5401a62 not found: ID does not exist"
Oct 30 19:56:53.312123 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:53.312093    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7"]
Oct 30 19:56:53.319167 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:53.319145    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f5qx7"]
Oct 30 19:56:53.330067 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:53.330041    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6"]
Oct 30 19:56:53.332367 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:53.332343    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qj2g6"]
Oct 30 19:56:54.643728 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:54.643693    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91978947-9c53-4a67-8f97-36ce17619c17" path="/var/lib/kubelet/pods/91978947-9c53-4a67-8f97-36ce17619c17/volumes"
Oct 30 19:56:54.644169 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:54.644153    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f2b4d7b-0014-49da-b5c2-0cf1944e8796" path="/var/lib/kubelet/pods/9f2b4d7b-0014-49da-b5c2-0cf1944e8796/volumes"
Oct 30 19:56:54.644540 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:56:54.644527    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba6f075a-2d9f-43f3-900f-2eced4621260" path="/var/lib/kubelet/pods/ba6f075a-2d9f-43f3-900f-2eced4621260/volumes"
Oct 30 19:56:54.977610 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:56:54.977564    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:57:17.401163 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:17.401128    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zgd28"]
Oct 30 19:57:17.403512 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:17.401547    5043 cpu_manager.go:414] "containerMap: removing container" podUID="ba6f075a-2d9f-43f3-900f-2eced4621260" containerName="istio-proxy"
Oct 30 19:57:17.403512 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:17.401559    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba6f075a-2d9f-43f3-900f-2eced4621260" containerName="istio-proxy"
Oct 30 19:57:17.403512 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:17.401571    5043 cpu_manager.go:414] "containerMap: removing container" podUID="91978947-9c53-4a67-8f97-36ce17619c17" containerName="istio-proxy"
Oct 30 19:57:17.403512 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:17.401576    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91978947-9c53-4a67-8f97-36ce17619c17" containerName="istio-proxy"
Oct 30 19:57:17.403512 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:17.401594    5043 cpu_manager.go:414] "containerMap: removing container" podUID="9f2b4d7b-0014-49da-b5c2-0cf1944e8796" containerName="istio-proxy"
Oct 30 19:57:17.403512 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:17.401598    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f2b4d7b-0014-49da-b5c2-0cf1944e8796" containerName="istio-proxy"
Oct 30 19:57:17.403512 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:17.401663    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f2b4d7b-0014-49da-b5c2-0cf1944e8796" containerName="istio-proxy"
Oct 30 19:57:17.403512 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:17.401676    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1ca4728-e019-4147-b9a3-d3f662d98e77" containerName="istio-proxy"
Oct 30 19:57:17.403512 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:17.401682    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba6f075a-2d9f-43f3-900f-2eced4621260" containerName="istio-proxy"
Oct 30 19:57:17.403512 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:17.401687    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="91978947-9c53-4a67-8f97-36ce17619c17" containerName="istio-proxy"
Oct 30 19:57:17.404630 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:17.404614    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zgd28"
Oct 30 19:57:17.420179 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:17.420144    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zgd28"]
Oct 30 19:57:17.483540 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:17.483507    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a958e54-98cc-4651-8c1b-dcb6b6207d1e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zgd28\" (UID: \"8a958e54-98cc-4651-8c1b-dcb6b6207d1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zgd28"
Oct 30 19:57:17.483540 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:17.483545    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a958e54-98cc-4651-8c1b-dcb6b6207d1e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zgd28\" (UID: \"8a958e54-98cc-4651-8c1b-dcb6b6207d1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zgd28"
Oct 30 19:57:17.483819 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:17.483570    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx257\" (UniqueName: \"kubernetes.io/projected/8a958e54-98cc-4651-8c1b-dcb6b6207d1e-kube-api-access-bx257\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zgd28\" (UID: \"8a958e54-98cc-4651-8c1b-dcb6b6207d1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zgd28"
Oct 30 19:57:17.483819 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:17.483655    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a958e54-98cc-4651-8c1b-dcb6b6207d1e-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zgd28\" (UID: \"8a958e54-98cc-4651-8c1b-dcb6b6207d1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zgd28"
Oct 30 19:57:17.483819 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:17.483689    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a958e54-98cc-4651-8c1b-dcb6b6207d1e-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zgd28\" (UID: \"8a958e54-98cc-4651-8c1b-dcb6b6207d1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zgd28"
Oct 30 19:57:17.483819 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:17.483722    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a958e54-98cc-4651-8c1b-dcb6b6207d1e-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zgd28\" (UID: \"8a958e54-98cc-4651-8c1b-dcb6b6207d1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zgd28"
Oct 30 19:57:17.483979 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:17.483846    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a958e54-98cc-4651-8c1b-dcb6b6207d1e-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zgd28\" (UID: \"8a958e54-98cc-4651-8c1b-dcb6b6207d1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zgd28"
Oct 30 19:57:17.483979 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:17.483872    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a958e54-98cc-4651-8c1b-dcb6b6207d1e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zgd28\" (UID: \"8a958e54-98cc-4651-8c1b-dcb6b6207d1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zgd28"
Oct 30 19:57:17.483979 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:17.483890    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a958e54-98cc-4651-8c1b-dcb6b6207d1e-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zgd28\" (UID: \"8a958e54-98cc-4651-8c1b-dcb6b6207d1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zgd28"
Oct 30 19:57:17.498357 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:17.498332    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-gt4zp\""
Oct 30 19:57:17.585459 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:17.585425    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a958e54-98cc-4651-8c1b-dcb6b6207d1e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zgd28\" (UID: \"8a958e54-98cc-4651-8c1b-dcb6b6207d1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zgd28"
Oct 30 19:57:17.585459 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:17.585462    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a958e54-98cc-4651-8c1b-dcb6b6207d1e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zgd28\" (UID: \"8a958e54-98cc-4651-8c1b-dcb6b6207d1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zgd28"
Oct 30 19:57:17.585715 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:17.585489    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx257\" (UniqueName: \"kubernetes.io/projected/8a958e54-98cc-4651-8c1b-dcb6b6207d1e-kube-api-access-bx257\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zgd28\" (UID: \"8a958e54-98cc-4651-8c1b-dcb6b6207d1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zgd28"
Oct 30 19:57:17.585715 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:17.585506    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a958e54-98cc-4651-8c1b-dcb6b6207d1e-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zgd28\" (UID: \"8a958e54-98cc-4651-8c1b-dcb6b6207d1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zgd28"
Oct 30 19:57:17.585715 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:17.585522    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a958e54-98cc-4651-8c1b-dcb6b6207d1e-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zgd28\" (UID: \"8a958e54-98cc-4651-8c1b-dcb6b6207d1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zgd28"
Oct 30 19:57:17.585715 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:17.585557    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a958e54-98cc-4651-8c1b-dcb6b6207d1e-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zgd28\" (UID: \"8a958e54-98cc-4651-8c1b-dcb6b6207d1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zgd28"
Oct 30 19:57:17.585715 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:17.585618    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a958e54-98cc-4651-8c1b-dcb6b6207d1e-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zgd28\" (UID: \"8a958e54-98cc-4651-8c1b-dcb6b6207d1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zgd28"
Oct 30 19:57:17.585715 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:17.585648    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a958e54-98cc-4651-8c1b-dcb6b6207d1e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zgd28\" (UID: \"8a958e54-98cc-4651-8c1b-dcb6b6207d1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zgd28"
Oct 30 19:57:17.585715 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:17.585679    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a958e54-98cc-4651-8c1b-dcb6b6207d1e-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zgd28\" (UID: \"8a958e54-98cc-4651-8c1b-dcb6b6207d1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zgd28"
Oct 30 19:57:17.586165 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:17.586044    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a958e54-98cc-4651-8c1b-dcb6b6207d1e-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zgd28\" (UID: \"8a958e54-98cc-4651-8c1b-dcb6b6207d1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zgd28"
Oct 30 19:57:17.586165 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:17.586100    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a958e54-98cc-4651-8c1b-dcb6b6207d1e-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zgd28\" (UID: \"8a958e54-98cc-4651-8c1b-dcb6b6207d1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zgd28"
Oct 30 19:57:17.586165 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:17.586158    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a958e54-98cc-4651-8c1b-dcb6b6207d1e-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zgd28\" (UID: \"8a958e54-98cc-4651-8c1b-dcb6b6207d1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zgd28"
Oct 30 19:57:17.586359 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:17.586340    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a958e54-98cc-4651-8c1b-dcb6b6207d1e-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zgd28\" (UID: \"8a958e54-98cc-4651-8c1b-dcb6b6207d1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zgd28"
Oct 30 19:57:17.586670 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:17.586646    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a958e54-98cc-4651-8c1b-dcb6b6207d1e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zgd28\" (UID: \"8a958e54-98cc-4651-8c1b-dcb6b6207d1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zgd28"
Oct 30 19:57:17.588085 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:17.588065    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a958e54-98cc-4651-8c1b-dcb6b6207d1e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zgd28\" (UID: \"8a958e54-98cc-4651-8c1b-dcb6b6207d1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zgd28"
Oct 30 19:57:17.588226 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:17.588204    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a958e54-98cc-4651-8c1b-dcb6b6207d1e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zgd28\" (UID: \"8a958e54-98cc-4651-8c1b-dcb6b6207d1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zgd28"
Oct 30 19:57:17.593708 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:17.593685    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a958e54-98cc-4651-8c1b-dcb6b6207d1e-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zgd28\" (UID: \"8a958e54-98cc-4651-8c1b-dcb6b6207d1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zgd28"
Oct 30 19:57:17.593945 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:17.593928    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx257\" (UniqueName: \"kubernetes.io/projected/8a958e54-98cc-4651-8c1b-dcb6b6207d1e-kube-api-access-bx257\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zgd28\" (UID: \"8a958e54-98cc-4651-8c1b-dcb6b6207d1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zgd28"
Oct 30 19:57:17.722813 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:17.722785    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zgd28"
Oct 30 19:57:18.011204 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:18.011178    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zgd28"]
Oct 30 19:57:18.013126 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:57:18.013095    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a958e54_98cc_4651_8c1b_dcb6b6207d1e.slice/crio-c643cc1b486816c39057a5eb1d50ccc9ba5694e57949ea1ec88b4fa6021db58f WatchSource:0}: Error finding container c643cc1b486816c39057a5eb1d50ccc9ba5694e57949ea1ec88b4fa6021db58f: Status 404 returned error can't find the container with id c643cc1b486816c39057a5eb1d50ccc9ba5694e57949ea1ec88b4fa6021db58f
Oct 30 19:57:18.015664 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:18.015632    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:57:18.015815 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:18.015707    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:57:18.015815 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:18.015781    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:57:18.403038 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:18.402949    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zgd28" event={"ID":"8a958e54-98cc-4651-8c1b-dcb6b6207d1e","Type":"ContainerStarted","Data":"bbcf9cfb19f68662ff4c0f41e4ee820f54539b256e186f2a96e3db65012d9a34"}
Oct 30 19:57:18.403038 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:18.403007    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zgd28" event={"ID":"8a958e54-98cc-4651-8c1b-dcb6b6207d1e","Type":"ContainerStarted","Data":"c643cc1b486816c39057a5eb1d50ccc9ba5694e57949ea1ec88b4fa6021db58f"}
Oct 30 19:57:18.425766 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:18.425700    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zgd28" podStartSLOduration=1.425686092 podStartE2EDuration="1.425686092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:18.423903254 +0000 UTC m=+2457.934963220" watchObservedRunningTime="2026-10-30 19:57:18.425686092 +0000 UTC m=+2457.936746056"
Oct 30 19:57:18.723619 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:18.723585    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zgd28"
Oct 30 19:57:19.725382 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:19.725346    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-zgd28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body=
Oct 30 19:57:19.725794 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:19.725411    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zgd28" podUID="8a958e54-98cc-4651-8c1b-dcb6b6207d1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused"
Oct 30 19:57:20.724006 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:20.723968    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-zgd28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body=
Oct 30 19:57:20.724243 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:20.724018    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zgd28" podUID="8a958e54-98cc-4651-8c1b-dcb6b6207d1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused"
Oct 30 19:57:21.724029 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:21.723993    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-zgd28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body=
Oct 30 19:57:21.724448 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:21.724047    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zgd28" podUID="8a958e54-98cc-4651-8c1b-dcb6b6207d1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused"
Oct 30 19:57:22.724124 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:22.724081    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-zgd28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body=
Oct 30 19:57:22.724636 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:22.724143    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zgd28" podUID="8a958e54-98cc-4651-8c1b-dcb6b6207d1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused"
Oct 30 19:57:23.723551 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:23.723512    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-zgd28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body=
Oct 30 19:57:23.723777 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:23.723562    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zgd28" podUID="8a958e54-98cc-4651-8c1b-dcb6b6207d1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused"
Oct 30 19:57:24.723945 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:24.723912    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-zgd28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body=
Oct 30 19:57:24.724332 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:24.723964    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zgd28" podUID="8a958e54-98cc-4651-8c1b-dcb6b6207d1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused"
Oct 30 19:57:24.983479 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:57:24.983391    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:57:26.583975 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:26.583935    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zgd28"
Oct 30 19:57:26.584563 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:26.584388    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zgd28"
Oct 30 19:57:26.586581 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:26.586552    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zgd28"
Oct 30 19:57:36.856492 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:36.856405    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9621d482-j9r44"]
Oct 30 19:57:36.859584 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:36.859559    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9621d482-j9r44"
Oct 30 19:57:36.864205 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:36.864175    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:57:36.865232 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:36.865208    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:57:36.865438 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:36.865416    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:57:36.865530 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:36.865506    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:57:36.868011 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:36.867985    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9621d482-j9r44"]
Oct 30 19:57:36.982569 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:36.982530    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rblff\" (UniqueName: \"kubernetes.io/projected/ae4a2f97-73b4-4c43-ae00-c47151d4c99a-kube-api-access-rblff\") pod \"maas-api-revoke-sub-9621d482-j9r44\" (UID: \"ae4a2f97-73b4-4c43-ae00-c47151d4c99a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9621d482-j9r44"
Oct 30 19:57:36.982804 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:36.982585    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae4a2f97-73b4-4c43-ae00-c47151d4c99a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9621d482-j9r44\" (UID: \"ae4a2f97-73b4-4c43-ae00-c47151d4c99a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9621d482-j9r44"
Oct 30 19:57:36.982804 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:36.982781    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae4a2f97-73b4-4c43-ae00-c47151d4c99a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9621d482-j9r44\" (UID: \"ae4a2f97-73b4-4c43-ae00-c47151d4c99a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9621d482-j9r44"
Oct 30 19:57:37.083782 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:37.083711    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rblff\" (UniqueName: \"kubernetes.io/projected/ae4a2f97-73b4-4c43-ae00-c47151d4c99a-kube-api-access-rblff\") pod \"maas-api-revoke-sub-9621d482-j9r44\" (UID: \"ae4a2f97-73b4-4c43-ae00-c47151d4c99a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9621d482-j9r44"
Oct 30 19:57:37.084004 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:37.083815    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae4a2f97-73b4-4c43-ae00-c47151d4c99a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9621d482-j9r44\" (UID: \"ae4a2f97-73b4-4c43-ae00-c47151d4c99a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9621d482-j9r44"
Oct 30 19:57:37.084004 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:37.083902    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae4a2f97-73b4-4c43-ae00-c47151d4c99a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9621d482-j9r44\" (UID: \"ae4a2f97-73b4-4c43-ae00-c47151d4c99a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9621d482-j9r44"
Oct 30 19:57:37.084454 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:37.084431    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae4a2f97-73b4-4c43-ae00-c47151d4c99a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9621d482-j9r44\" (UID: \"ae4a2f97-73b4-4c43-ae00-c47151d4c99a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9621d482-j9r44"
Oct 30 19:57:37.086489 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:37.086466    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae4a2f97-73b4-4c43-ae00-c47151d4c99a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9621d482-j9r44\" (UID: \"ae4a2f97-73b4-4c43-ae00-c47151d4c99a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9621d482-j9r44"
Oct 30 19:57:37.091201 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:37.091180    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rblff\" (UniqueName: \"kubernetes.io/projected/ae4a2f97-73b4-4c43-ae00-c47151d4c99a-kube-api-access-rblff\") pod \"maas-api-revoke-sub-9621d482-j9r44\" (UID: \"ae4a2f97-73b4-4c43-ae00-c47151d4c99a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9621d482-j9r44"
Oct 30 19:57:37.177302 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:37.177180    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9621d482-j9r44"
Oct 30 19:57:37.456191 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:37.456153    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9621d482-j9r44"]
Oct 30 19:57:37.459032 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:57:37.459001    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae4a2f97_73b4_4c43_ae00_c47151d4c99a.slice/crio-8df90c53d3947b8a84fa556cbcd0bb681f575941c87c33c8fd91d338dca0e769 WatchSource:0}: Error finding container 8df90c53d3947b8a84fa556cbcd0bb681f575941c87c33c8fd91d338dca0e769: Status 404 returned error can't find the container with id 8df90c53d3947b8a84fa556cbcd0bb681f575941c87c33c8fd91d338dca0e769
Oct 30 19:57:37.515661 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:37.515626    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9621d482-j9r44" event={"ID":"ae4a2f97-73b4-4c43-ae00-c47151d4c99a","Type":"ContainerStarted","Data":"8df90c53d3947b8a84fa556cbcd0bb681f575941c87c33c8fd91d338dca0e769"}
Oct 30 19:57:38.522014 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:38.521977    5043 generic.go:356] "Generic (PLEG): container finished" podID="ae4a2f97-73b4-4c43-ae00-c47151d4c99a" containerID="1d4bf6b55754c84f31d759e94b57aa39e1e36181f55ccf235fbc3e102760939c" exitCode=0
Oct 30 19:57:38.522480 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:38.522058    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9621d482-j9r44" event={"ID":"ae4a2f97-73b4-4c43-ae00-c47151d4c99a","Type":"ContainerDied","Data":"1d4bf6b55754c84f31d759e94b57aa39e1e36181f55ccf235fbc3e102760939c"}
Oct 30 19:57:39.676768 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:39.676715    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9621d482-j9r44"
Oct 30 19:57:39.808607 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:39.808522    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae4a2f97-73b4-4c43-ae00-c47151d4c99a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae4a2f97-73b4-4c43-ae00-c47151d4c99a-maas-api-xks-ca\") pod \"ae4a2f97-73b4-4c43-ae00-c47151d4c99a\" (UID: \"ae4a2f97-73b4-4c43-ae00-c47151d4c99a\") "
Oct 30 19:57:39.808866 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:39.808683    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae4a2f97-73b4-4c43-ae00-c47151d4c99a-kube-api-access-rblff\" (UniqueName: \"kubernetes.io/projected/ae4a2f97-73b4-4c43-ae00-c47151d4c99a-kube-api-access-rblff\") pod \"ae4a2f97-73b4-4c43-ae00-c47151d4c99a\" (UID: \"ae4a2f97-73b4-4c43-ae00-c47151d4c99a\") "
Oct 30 19:57:39.808866 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:39.808811    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae4a2f97-73b4-4c43-ae00-c47151d4c99a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae4a2f97-73b4-4c43-ae00-c47151d4c99a-maas-api-openshift-ca\") pod \"ae4a2f97-73b4-4c43-ae00-c47151d4c99a\" (UID: \"ae4a2f97-73b4-4c43-ae00-c47151d4c99a\") "
Oct 30 19:57:39.809192 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:39.809169    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae4a2f97-73b4-4c43-ae00-c47151d4c99a-maas-api-openshift-ca" pod "ae4a2f97-73b4-4c43-ae00-c47151d4c99a" (UID: "ae4a2f97-73b4-4c43-ae00-c47151d4c99a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:57:39.811064 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:39.811036    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae4a2f97-73b4-4c43-ae00-c47151d4c99a-maas-api-xks-ca" pod "ae4a2f97-73b4-4c43-ae00-c47151d4c99a" (UID: "ae4a2f97-73b4-4c43-ae00-c47151d4c99a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:57:39.811161 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:39.811101    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae4a2f97-73b4-4c43-ae00-c47151d4c99a-kube-api-access-rblff" pod "ae4a2f97-73b4-4c43-ae00-c47151d4c99a" (UID: "ae4a2f97-73b4-4c43-ae00-c47151d4c99a"). InnerVolumeSpecName "kube-api-access-rblff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:57:39.909992 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:39.909947    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae4a2f97-73b4-4c43-ae00-c47151d4c99a-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:39.909992 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:39.909985    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae4a2f97-73b4-4c43-ae00-c47151d4c99a-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:39.909992 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:39.910002    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rblff\" (UniqueName: \"kubernetes.io/projected/ae4a2f97-73b4-4c43-ae00-c47151d4c99a-kube-api-access-rblff\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:40.532844 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:40.532809    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9621d482-j9r44" event={"ID":"ae4a2f97-73b4-4c43-ae00-c47151d4c99a","Type":"ContainerDied","Data":"8df90c53d3947b8a84fa556cbcd0bb681f575941c87c33c8fd91d338dca0e769"}
Oct 30 19:57:40.532844 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:40.532830    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9621d482-j9r44"
Oct 30 19:57:40.533071 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:40.532869    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8df90c53d3947b8a84fa556cbcd0bb681f575941c87c33c8fd91d338dca0e769"
Oct 30 19:57:47.851671 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:47.851632    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-98b568-cg2jc"]
Oct 30 19:57:47.852270 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:47.852109    5043 cpu_manager.go:414] "containerMap: removing container" podUID="ae4a2f97-73b4-4c43-ae00-c47151d4c99a" containerName="revoke-keys"
Oct 30 19:57:47.852270 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:47.852121    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae4a2f97-73b4-4c43-ae00-c47151d4c99a" containerName="revoke-keys"
Oct 30 19:57:47.852270 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:47.852183    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae4a2f97-73b4-4c43-ae00-c47151d4c99a" containerName="revoke-keys"
Oct 30 19:57:47.855216 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:47.855201    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-98b568-cg2jc"
Oct 30 19:57:47.860867 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:47.860726    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:57:47.860867 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:47.860842    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:57:47.861557 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:47.861535    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 19:57:47.862396 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:47.862365    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:57:47.864840 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:47.864786    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-98b568-cg2jc"]
Oct 30 19:57:47.989559 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:47.989517    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5e81249-b39a-4c88-b8ed-6b5d19ce99fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-98b568-cg2jc\" (UID: \"e5e81249-b39a-4c88-b8ed-6b5d19ce99fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-98b568-cg2jc"
Oct 30 19:57:47.989784 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:47.989665    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqrc4\" (UniqueName: \"kubernetes.io/projected/e5e81249-b39a-4c88-b8ed-6b5d19ce99fe-kube-api-access-nqrc4\") pod \"maas-api-revoke-keys-e2e-worker-single-main-98b568-cg2jc\" (UID: \"e5e81249-b39a-4c88-b8ed-6b5d19ce99fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-98b568-cg2jc"
Oct 30 19:57:47.989784 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:47.989725    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5e81249-b39a-4c88-b8ed-6b5d19ce99fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-98b568-cg2jc\" (UID: \"e5e81249-b39a-4c88-b8ed-6b5d19ce99fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-98b568-cg2jc"
Oct 30 19:57:48.090907 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:48.090868    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqrc4\" (UniqueName: \"kubernetes.io/projected/e5e81249-b39a-4c88-b8ed-6b5d19ce99fe-kube-api-access-nqrc4\") pod \"maas-api-revoke-keys-e2e-worker-single-main-98b568-cg2jc\" (UID: \"e5e81249-b39a-4c88-b8ed-6b5d19ce99fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-98b568-cg2jc"
Oct 30 19:57:48.090907 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:48.090912    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5e81249-b39a-4c88-b8ed-6b5d19ce99fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-98b568-cg2jc\" (UID: \"e5e81249-b39a-4c88-b8ed-6b5d19ce99fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-98b568-cg2jc"
Oct 30 19:57:48.091174 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:48.090975    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5e81249-b39a-4c88-b8ed-6b5d19ce99fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-98b568-cg2jc\" (UID: \"e5e81249-b39a-4c88-b8ed-6b5d19ce99fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-98b568-cg2jc"
Oct 30 19:57:48.091693 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:48.091667    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5e81249-b39a-4c88-b8ed-6b5d19ce99fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-98b568-cg2jc\" (UID: \"e5e81249-b39a-4c88-b8ed-6b5d19ce99fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-98b568-cg2jc"
Oct 30 19:57:48.093868 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:48.093839    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5e81249-b39a-4c88-b8ed-6b5d19ce99fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-98b568-cg2jc\" (UID: \"e5e81249-b39a-4c88-b8ed-6b5d19ce99fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-98b568-cg2jc"
Oct 30 19:57:48.100914 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:48.100883    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqrc4\" (UniqueName: \"kubernetes.io/projected/e5e81249-b39a-4c88-b8ed-6b5d19ce99fe-kube-api-access-nqrc4\") pod \"maas-api-revoke-keys-e2e-worker-single-main-98b568-cg2jc\" (UID: \"e5e81249-b39a-4c88-b8ed-6b5d19ce99fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-98b568-cg2jc"
Oct 30 19:57:48.171527 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:48.171429    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-98b568-cg2jc"
Oct 30 19:57:48.464547 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:48.458589    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-98b568-cg2jc"]
Oct 30 19:57:48.576367 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:48.576329    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-98b568-cg2jc" event={"ID":"e5e81249-b39a-4c88-b8ed-6b5d19ce99fe","Type":"ContainerStarted","Data":"db002e3b255676e75dff6c57fed6612686090aaeb985c5fde5cc7a3483a3487b"}
Oct 30 19:57:49.583617 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:49.583578    5043 generic.go:356] "Generic (PLEG): container finished" podID="e5e81249-b39a-4c88-b8ed-6b5d19ce99fe" containerID="125f8b0dc6a5042fa13e42879e6c1784bb3cfcdfb2d00d1c152046801a80411f" exitCode=0
Oct 30 19:57:49.584054 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:49.583672    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-98b568-cg2jc" event={"ID":"e5e81249-b39a-4c88-b8ed-6b5d19ce99fe","Type":"ContainerDied","Data":"125f8b0dc6a5042fa13e42879e6c1784bb3cfcdfb2d00d1c152046801a80411f"}
Oct 30 19:57:50.733640 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:50.733614    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-98b568-cg2jc"
Oct 30 19:57:50.814279 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:50.814243    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5e81249-b39a-4c88-b8ed-6b5d19ce99fe-kube-api-access-nqrc4\" (UniqueName: \"kubernetes.io/projected/e5e81249-b39a-4c88-b8ed-6b5d19ce99fe-kube-api-access-nqrc4\") pod \"e5e81249-b39a-4c88-b8ed-6b5d19ce99fe\" (UID: \"e5e81249-b39a-4c88-b8ed-6b5d19ce99fe\") "
Oct 30 19:57:50.814456 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:50.814401    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5e81249-b39a-4c88-b8ed-6b5d19ce99fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5e81249-b39a-4c88-b8ed-6b5d19ce99fe-maas-api-xks-ca\") pod \"e5e81249-b39a-4c88-b8ed-6b5d19ce99fe\" (UID: \"e5e81249-b39a-4c88-b8ed-6b5d19ce99fe\") "
Oct 30 19:57:50.814456 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:50.814425    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5e81249-b39a-4c88-b8ed-6b5d19ce99fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5e81249-b39a-4c88-b8ed-6b5d19ce99fe-maas-api-openshift-ca\") pod \"e5e81249-b39a-4c88-b8ed-6b5d19ce99fe\" (UID: \"e5e81249-b39a-4c88-b8ed-6b5d19ce99fe\") "
Oct 30 19:57:50.814879 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:50.814852    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5e81249-b39a-4c88-b8ed-6b5d19ce99fe-maas-api-openshift-ca" pod "e5e81249-b39a-4c88-b8ed-6b5d19ce99fe" (UID: "e5e81249-b39a-4c88-b8ed-6b5d19ce99fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:57:50.816552 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:50.816529    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5e81249-b39a-4c88-b8ed-6b5d19ce99fe-maas-api-xks-ca" pod "e5e81249-b39a-4c88-b8ed-6b5d19ce99fe" (UID: "e5e81249-b39a-4c88-b8ed-6b5d19ce99fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:57:50.816654 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:50.816637    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5e81249-b39a-4c88-b8ed-6b5d19ce99fe-kube-api-access-nqrc4" pod "e5e81249-b39a-4c88-b8ed-6b5d19ce99fe" (UID: "e5e81249-b39a-4c88-b8ed-6b5d19ce99fe"). InnerVolumeSpecName "kube-api-access-nqrc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:57:50.916101 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:50.916019    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5e81249-b39a-4c88-b8ed-6b5d19ce99fe-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:50.916101 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:50.916048    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5e81249-b39a-4c88-b8ed-6b5d19ce99fe-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:50.916101 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:50.916058    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqrc4\" (UniqueName: \"kubernetes.io/projected/e5e81249-b39a-4c88-b8ed-6b5d19ce99fe-kube-api-access-nqrc4\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:51.596602 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:51.596570    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-98b568-cg2jc"
Oct 30 19:57:51.596851 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:51.596565    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-98b568-cg2jc" event={"ID":"e5e81249-b39a-4c88-b8ed-6b5d19ce99fe","Type":"ContainerDied","Data":"db002e3b255676e75dff6c57fed6612686090aaeb985c5fde5cc7a3483a3487b"}
Oct 30 19:57:51.596851 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:51.596714    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db002e3b255676e75dff6c57fed6612686090aaeb985c5fde5cc7a3483a3487b"
Oct 30 19:57:54.976297 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:57:54.976261    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:57:58.755678 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:58.755596    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-98b568-cg2jc"]
Oct 30 19:57:58.757820 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:57:58.757793    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-98b568-cg2jc"]
Oct 30 19:58:00.643845 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:00.643811    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5e81249-b39a-4c88-b8ed-6b5d19ce99fe" path="/var/lib/kubelet/pods/e5e81249-b39a-4c88-b8ed-6b5d19ce99fe/volumes"
Oct 30 19:58:24.976344 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:58:24.976305    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:58:25.820469 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:25.820438    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sdss4"]
Oct 30 19:58:25.820678 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:25.820656    5043 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sdss4" podUID="3929b4ec-4986-41a3-9fd0-929b3d254cb9" containerName="manager" containerID="cri-o://35a45a169678be98decd092dcd96ff9cf4a5fc83b62e3b15896e52726a524865" gracePeriod=10
Oct 30 19:58:26.093447 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:26.093424    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sdss4"
Oct 30 19:58:26.158550 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:26.158517    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3929b4ec-4986-41a3-9fd0-929b3d254cb9-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3929b4ec-4986-41a3-9fd0-929b3d254cb9-extensions-socket-volume\") pod \"3929b4ec-4986-41a3-9fd0-929b3d254cb9\" (UID: \"3929b4ec-4986-41a3-9fd0-929b3d254cb9\") "
Oct 30 19:58:26.158725 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:26.158662    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3929b4ec-4986-41a3-9fd0-929b3d254cb9-kube-api-access-hl78s\" (UniqueName: \"kubernetes.io/projected/3929b4ec-4986-41a3-9fd0-929b3d254cb9-kube-api-access-hl78s\") pod \"3929b4ec-4986-41a3-9fd0-929b3d254cb9\" (UID: \"3929b4ec-4986-41a3-9fd0-929b3d254cb9\") "
Oct 30 19:58:26.158920 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:26.158898    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3929b4ec-4986-41a3-9fd0-929b3d254cb9-extensions-socket-volume" pod "3929b4ec-4986-41a3-9fd0-929b3d254cb9" (UID: "3929b4ec-4986-41a3-9fd0-929b3d254cb9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:26.159038 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:26.159015    5043 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3929b4ec-4986-41a3-9fd0-929b3d254cb9-extensions-socket-volume\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:58:26.161042 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:26.161011    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3929b4ec-4986-41a3-9fd0-929b3d254cb9-kube-api-access-hl78s" pod "3929b4ec-4986-41a3-9fd0-929b3d254cb9" (UID: "3929b4ec-4986-41a3-9fd0-929b3d254cb9"). InnerVolumeSpecName "kube-api-access-hl78s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:58:26.260053 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:26.260021    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl78s\" (UniqueName: \"kubernetes.io/projected/3929b4ec-4986-41a3-9fd0-929b3d254cb9-kube-api-access-hl78s\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:58:26.781593 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:26.781558    5043 generic.go:356] "Generic (PLEG): container finished" podID="3929b4ec-4986-41a3-9fd0-929b3d254cb9" containerID="35a45a169678be98decd092dcd96ff9cf4a5fc83b62e3b15896e52726a524865" exitCode=0
Oct 30 19:58:26.781819 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:26.781607    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sdss4"
Oct 30 19:58:26.781819 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:26.781640    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sdss4" event={"ID":"3929b4ec-4986-41a3-9fd0-929b3d254cb9","Type":"ContainerDied","Data":"35a45a169678be98decd092dcd96ff9cf4a5fc83b62e3b15896e52726a524865"}
Oct 30 19:58:26.781819 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:26.781694    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sdss4" event={"ID":"3929b4ec-4986-41a3-9fd0-929b3d254cb9","Type":"ContainerDied","Data":"08e394b96a67b62fd615af438975f9572a5c707c9a2e5228f020b830c93ccbc8"}
Oct 30 19:58:26.781819 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:26.781715    5043 scope.go:122] "RemoveContainer" containerID="35a45a169678be98decd092dcd96ff9cf4a5fc83b62e3b15896e52726a524865"
Oct 30 19:58:26.794006 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:26.793990    5043 scope.go:122] "RemoveContainer" containerID="35a45a169678be98decd092dcd96ff9cf4a5fc83b62e3b15896e52726a524865"
Oct 30 19:58:26.794308 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:58:26.794288    5043 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35a45a169678be98decd092dcd96ff9cf4a5fc83b62e3b15896e52726a524865\": container with ID starting with 35a45a169678be98decd092dcd96ff9cf4a5fc83b62e3b15896e52726a524865 not found: ID does not exist" containerID="35a45a169678be98decd092dcd96ff9cf4a5fc83b62e3b15896e52726a524865"
Oct 30 19:58:26.794357 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:26.794319    5043 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35a45a169678be98decd092dcd96ff9cf4a5fc83b62e3b15896e52726a524865"} err="failed to get container status \"35a45a169678be98decd092dcd96ff9cf4a5fc83b62e3b15896e52726a524865\": rpc error: code = NotFound desc = could not find container \"35a45a169678be98decd092dcd96ff9cf4a5fc83b62e3b15896e52726a524865\": container with ID starting with 35a45a169678be98decd092dcd96ff9cf4a5fc83b62e3b15896e52726a524865 not found: ID does not exist"
Oct 30 19:58:26.800489 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:26.800467    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sdss4"]
Oct 30 19:58:26.806375 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:26.806351    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sdss4"]
Oct 30 19:58:27.084049 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:27.083997    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-sdss4" podUID="3929b4ec-4986-41a3-9fd0-929b3d254cb9" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:58:28.643965 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:28.643934    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3929b4ec-4986-41a3-9fd0-929b3d254cb9" path="/var/lib/kubelet/pods/3929b4ec-4986-41a3-9fd0-929b3d254cb9/volumes"
Oct 30 19:58:47.544246 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:47.544208    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57"]
Oct 30 19:58:47.547058 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:47.545700    5043 cpu_manager.go:414] "containerMap: removing container" podUID="e5e81249-b39a-4c88-b8ed-6b5d19ce99fe" containerName="revoke-keys"
Oct 30 19:58:47.547058 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:47.545726    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5e81249-b39a-4c88-b8ed-6b5d19ce99fe" containerName="revoke-keys"
Oct 30 19:58:47.547058 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:47.545783    5043 cpu_manager.go:414] "containerMap: removing container" podUID="3929b4ec-4986-41a3-9fd0-929b3d254cb9" containerName="manager"
Oct 30 19:58:47.547058 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:47.545792    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3929b4ec-4986-41a3-9fd0-929b3d254cb9" containerName="manager"
Oct 30 19:58:47.547058 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:47.545996    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5e81249-b39a-4c88-b8ed-6b5d19ce99fe" containerName="revoke-keys"
Oct 30 19:58:47.547058 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:47.546016    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="3929b4ec-4986-41a3-9fd0-929b3d254cb9" containerName="manager"
Oct 30 19:58:47.552785 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:47.552762    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57"
Oct 30 19:58:47.561560 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:47.560879    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc14aeeb-67bf-438e-870e-595e528ba5c4-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jrm57\" (UID: \"cc14aeeb-67bf-438e-870e-595e528ba5c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57"
Oct 30 19:58:47.561560 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:47.560945    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfgt7\" (UniqueName: \"kubernetes.io/projected/cc14aeeb-67bf-438e-870e-595e528ba5c4-kube-api-access-jfgt7\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jrm57\" (UID: \"cc14aeeb-67bf-438e-870e-595e528ba5c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57"
Oct 30 19:58:47.561560 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:47.561031    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc14aeeb-67bf-438e-870e-595e528ba5c4-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jrm57\" (UID: \"cc14aeeb-67bf-438e-870e-595e528ba5c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57"
Oct 30 19:58:47.561560 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:47.561131    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc14aeeb-67bf-438e-870e-595e528ba5c4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jrm57\" (UID: \"cc14aeeb-67bf-438e-870e-595e528ba5c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57"
Oct 30 19:58:47.561560 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:47.561160    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc14aeeb-67bf-438e-870e-595e528ba5c4-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jrm57\" (UID: \"cc14aeeb-67bf-438e-870e-595e528ba5c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57"
Oct 30 19:58:47.561560 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:47.561188    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc14aeeb-67bf-438e-870e-595e528ba5c4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jrm57\" (UID: \"cc14aeeb-67bf-438e-870e-595e528ba5c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57"
Oct 30 19:58:47.561560 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:47.561235    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc14aeeb-67bf-438e-870e-595e528ba5c4-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jrm57\" (UID: \"cc14aeeb-67bf-438e-870e-595e528ba5c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57"
Oct 30 19:58:47.561560 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:47.561263    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc14aeeb-67bf-438e-870e-595e528ba5c4-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jrm57\" (UID: \"cc14aeeb-67bf-438e-870e-595e528ba5c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57"
Oct 30 19:58:47.561560 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:47.561292    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc14aeeb-67bf-438e-870e-595e528ba5c4-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jrm57\" (UID: \"cc14aeeb-67bf-438e-870e-595e528ba5c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57"
Oct 30 19:58:47.566588 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:47.566560    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57"]
Oct 30 19:58:47.662877 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:47.662841    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc14aeeb-67bf-438e-870e-595e528ba5c4-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jrm57\" (UID: \"cc14aeeb-67bf-438e-870e-595e528ba5c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57"
Oct 30 19:58:47.662877 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:47.662886    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfgt7\" (UniqueName: \"kubernetes.io/projected/cc14aeeb-67bf-438e-870e-595e528ba5c4-kube-api-access-jfgt7\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jrm57\" (UID: \"cc14aeeb-67bf-438e-870e-595e528ba5c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57"
Oct 30 19:58:47.663167 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:47.662921    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc14aeeb-67bf-438e-870e-595e528ba5c4-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jrm57\" (UID: \"cc14aeeb-67bf-438e-870e-595e528ba5c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57"
Oct 30 19:58:47.663167 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:47.663008    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc14aeeb-67bf-438e-870e-595e528ba5c4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jrm57\" (UID: \"cc14aeeb-67bf-438e-870e-595e528ba5c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57"
Oct 30 19:58:47.663167 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:47.663043    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc14aeeb-67bf-438e-870e-595e528ba5c4-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jrm57\" (UID: \"cc14aeeb-67bf-438e-870e-595e528ba5c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57"
Oct 30 19:58:47.663167 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:47.663071    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc14aeeb-67bf-438e-870e-595e528ba5c4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jrm57\" (UID: \"cc14aeeb-67bf-438e-870e-595e528ba5c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57"
Oct 30 19:58:47.663167 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:47.663121    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc14aeeb-67bf-438e-870e-595e528ba5c4-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jrm57\" (UID: \"cc14aeeb-67bf-438e-870e-595e528ba5c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57"
Oct 30 19:58:47.663167 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:47.663151    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc14aeeb-67bf-438e-870e-595e528ba5c4-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jrm57\" (UID: \"cc14aeeb-67bf-438e-870e-595e528ba5c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57"
Oct 30 19:58:47.663527 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:47.663210    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc14aeeb-67bf-438e-870e-595e528ba5c4-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jrm57\" (UID: \"cc14aeeb-67bf-438e-870e-595e528ba5c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57"
Oct 30 19:58:47.663527 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:47.663329    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc14aeeb-67bf-438e-870e-595e528ba5c4-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jrm57\" (UID: \"cc14aeeb-67bf-438e-870e-595e528ba5c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57"
Oct 30 19:58:47.663660 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:47.663633    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc14aeeb-67bf-438e-870e-595e528ba5c4-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jrm57\" (UID: \"cc14aeeb-67bf-438e-870e-595e528ba5c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57"
Oct 30 19:58:47.663724 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:47.663693    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc14aeeb-67bf-438e-870e-595e528ba5c4-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jrm57\" (UID: \"cc14aeeb-67bf-438e-870e-595e528ba5c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57"
Oct 30 19:58:47.663830 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:47.663786    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc14aeeb-67bf-438e-870e-595e528ba5c4-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jrm57\" (UID: \"cc14aeeb-67bf-438e-870e-595e528ba5c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57"
Oct 30 19:58:47.663938 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:47.663918    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc14aeeb-67bf-438e-870e-595e528ba5c4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jrm57\" (UID: \"cc14aeeb-67bf-438e-870e-595e528ba5c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57"
Oct 30 19:58:47.665662 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:47.665640    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc14aeeb-67bf-438e-870e-595e528ba5c4-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jrm57\" (UID: \"cc14aeeb-67bf-438e-870e-595e528ba5c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57"
Oct 30 19:58:47.665932 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:47.665910    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc14aeeb-67bf-438e-870e-595e528ba5c4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jrm57\" (UID: \"cc14aeeb-67bf-438e-870e-595e528ba5c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57"
Oct 30 19:58:47.671825 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:47.671795    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc14aeeb-67bf-438e-870e-595e528ba5c4-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jrm57\" (UID: \"cc14aeeb-67bf-438e-870e-595e528ba5c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57"
Oct 30 19:58:47.671934 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:47.671859    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfgt7\" (UniqueName: \"kubernetes.io/projected/cc14aeeb-67bf-438e-870e-595e528ba5c4-kube-api-access-jfgt7\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jrm57\" (UID: \"cc14aeeb-67bf-438e-870e-595e528ba5c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57"
Oct 30 19:58:47.874151 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:47.874055    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57"
Oct 30 19:58:48.157178 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:48.157040    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57"]
Oct 30 19:58:48.159683 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:58:48.159658    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc14aeeb_67bf_438e_870e_595e528ba5c4.slice/crio-7cf1431f770184cba1e8caa532dd39b389f50759e8f7d0d7c50f8198f5c83cd3 WatchSource:0}: Error finding container 7cf1431f770184cba1e8caa532dd39b389f50759e8f7d0d7c50f8198f5c83cd3: Status 404 returned error can't find the container with id 7cf1431f770184cba1e8caa532dd39b389f50759e8f7d0d7c50f8198f5c83cd3
Oct 30 19:58:48.162821 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:48.162783    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:58:48.162927 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:48.162862    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:58:48.162927 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:48.162894    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:58:48.897047 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:48.897014    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57" event={"ID":"cc14aeeb-67bf-438e-870e-595e528ba5c4","Type":"ContainerStarted","Data":"6057c59d389531f0c1c1307639910671ac8ff23259caeae62c679ec68c3c1a61"}
Oct 30 19:58:48.897484 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:48.897073    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57" event={"ID":"cc14aeeb-67bf-438e-870e-595e528ba5c4","Type":"ContainerStarted","Data":"7cf1431f770184cba1e8caa532dd39b389f50759e8f7d0d7c50f8198f5c83cd3"}
Oct 30 19:58:48.921147 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:48.921090    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57" podStartSLOduration=1.921076674 podStartE2EDuration="1.921076674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:48.918331741 +0000 UTC m=+2548.429391727" watchObservedRunningTime="2026-10-30 19:58:48.921076674 +0000 UTC m=+2548.432136639"
Oct 30 19:58:49.874582 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:49.874548    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57"
Oct 30 19:58:49.879649 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:49.879626    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57"
Oct 30 19:58:49.902887 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:49.902853    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57"
Oct 30 19:58:49.903835 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:58:49.903809    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57"
Oct 30 19:58:54.975728 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:58:54.975688    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:59:02.548715 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:02.548624    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp"]
Oct 30 19:59:02.552755 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:02.552723    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp"
Oct 30 19:59:02.570170 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:02.570141    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp"]
Oct 30 19:59:02.598645 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:02.598604    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7r2x\" (UniqueName: \"kubernetes.io/projected/4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21-kube-api-access-v7r2x\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ftjzp\" (UID: \"4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp"
Oct 30 19:59:02.598865 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:02.598660    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ftjzp\" (UID: \"4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp"
Oct 30 19:59:02.598865 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:02.598719    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ftjzp\" (UID: \"4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp"
Oct 30 19:59:02.598865 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:02.598839    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ftjzp\" (UID: \"4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp"
Oct 30 19:59:02.599144 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:02.598865    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ftjzp\" (UID: \"4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp"
Oct 30 19:59:02.599144 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:02.598894    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ftjzp\" (UID: \"4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp"
Oct 30 19:59:02.599144 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:02.598949    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ftjzp\" (UID: \"4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp"
Oct 30 19:59:02.599144 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:02.598971    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ftjzp\" (UID: \"4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp"
Oct 30 19:59:02.599144 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:02.598998    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ftjzp\" (UID: \"4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp"
Oct 30 19:59:02.700478 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:02.700445    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ftjzp\" (UID: \"4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp"
Oct 30 19:59:02.700478 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:02.700482    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ftjzp\" (UID: \"4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp"
Oct 30 19:59:02.700760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:02.700513    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ftjzp\" (UID: \"4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp"
Oct 30 19:59:02.700760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:02.700542    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ftjzp\" (UID: \"4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp"
Oct 30 19:59:02.700760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:02.700579    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ftjzp\" (UID: \"4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp"
Oct 30 19:59:02.700760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:02.700618    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7r2x\" (UniqueName: \"kubernetes.io/projected/4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21-kube-api-access-v7r2x\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ftjzp\" (UID: \"4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp"
Oct 30 19:59:02.700760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:02.700674    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ftjzp\" (UID: \"4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp"
Oct 30 19:59:02.700760 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:02.700700    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ftjzp\" (UID: \"4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp"
Oct 30 19:59:02.701104 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:02.700822    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ftjzp\" (UID: \"4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp"
Oct 30 19:59:02.701104 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:02.700926    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ftjzp\" (UID: \"4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp"
Oct 30 19:59:02.701104 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:02.701018    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ftjzp\" (UID: \"4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp"
Oct 30 19:59:02.701275 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:02.701250    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ftjzp\" (UID: \"4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp"
Oct 30 19:59:02.701380 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:02.701358    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ftjzp\" (UID: \"4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp"
Oct 30 19:59:02.701532 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:02.701515    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ftjzp\" (UID: \"4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp"
Oct 30 19:59:02.703140 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:02.703116    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ftjzp\" (UID: \"4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp"
Oct 30 19:59:02.703346 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:02.703329    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ftjzp\" (UID: \"4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp"
Oct 30 19:59:02.709819 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:02.709798    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7r2x\" (UniqueName: \"kubernetes.io/projected/4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21-kube-api-access-v7r2x\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ftjzp\" (UID: \"4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp"
Oct 30 19:59:02.709920 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:02.709867    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ftjzp\" (UID: \"4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp"
Oct 30 19:59:02.872973 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:02.872876    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp"
Oct 30 19:59:03.157403 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:03.157376    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp"]
Oct 30 19:59:03.159633 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:59:03.159610    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d17d9d4_53d0_4f8e_87a1_3b43f94d2b21.slice/crio-32ca4cd93f6060d75697bd25f70ed7a8e698c7eb8af2ac878c27c5505a1e2868 WatchSource:0}: Error finding container 32ca4cd93f6060d75697bd25f70ed7a8e698c7eb8af2ac878c27c5505a1e2868: Status 404 returned error can't find the container with id 32ca4cd93f6060d75697bd25f70ed7a8e698c7eb8af2ac878c27c5505a1e2868
Oct 30 19:59:03.161877 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:03.161846    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:59:03.161986 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:03.161929    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:59:03.161986 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:03.161973    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 19:59:03.974362 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:03.974327    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp" event={"ID":"4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21","Type":"ContainerStarted","Data":"22d0aebc9d4cec4dde38bceefc8fd9319937d384bf0d59b3e503cb49a8677fbe"}
Oct 30 19:59:03.974786 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:03.974389    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp" event={"ID":"4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21","Type":"ContainerStarted","Data":"32ca4cd93f6060d75697bd25f70ed7a8e698c7eb8af2ac878c27c5505a1e2868"}
Oct 30 19:59:03.995411 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:03.995363    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp" podStartSLOduration=1.995348444 podStartE2EDuration="1.995348444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:03.993335849 +0000 UTC m=+2563.504395812" watchObservedRunningTime="2026-10-30 19:59:03.995348444 +0000 UTC m=+2563.506408409"
Oct 30 19:59:04.873983 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:04.873947    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp"
Oct 30 19:59:04.879455 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:04.879430    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp"
Oct 30 19:59:04.980574 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:04.980545    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp"
Oct 30 19:59:04.981565 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:04.981541    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp"
Oct 30 19:59:24.975589 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:59:24.975553    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 19:59:33.368190 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:33.368158    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4268w"]
Oct 30 19:59:33.373053 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:33.373030    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4268w"
Oct 30 19:59:33.378909 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:33.378887    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-p995g\""
Oct 30 19:59:33.385891 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:33.385865    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4268w"]
Oct 30 19:59:33.400708 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:33.400679    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1d0d721e-fbdc-4f18-88a2-094f7e42f9f8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-4268w\" (UID: \"1d0d721e-fbdc-4f18-88a2-094f7e42f9f8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4268w"
Oct 30 19:59:33.400852 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:33.400717    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wldw\" (UniqueName: \"kubernetes.io/projected/1d0d721e-fbdc-4f18-88a2-094f7e42f9f8-kube-api-access-8wldw\") pod \"kuadrant-operator-controller-manager-678d87766f-4268w\" (UID: \"1d0d721e-fbdc-4f18-88a2-094f7e42f9f8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4268w"
Oct 30 19:59:33.501570 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:33.501535    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1d0d721e-fbdc-4f18-88a2-094f7e42f9f8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-4268w\" (UID: \"1d0d721e-fbdc-4f18-88a2-094f7e42f9f8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4268w"
Oct 30 19:59:33.501570 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:33.501574    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wldw\" (UniqueName: \"kubernetes.io/projected/1d0d721e-fbdc-4f18-88a2-094f7e42f9f8-kube-api-access-8wldw\") pod \"kuadrant-operator-controller-manager-678d87766f-4268w\" (UID: \"1d0d721e-fbdc-4f18-88a2-094f7e42f9f8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4268w"
Oct 30 19:59:33.501938 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:33.501918    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1d0d721e-fbdc-4f18-88a2-094f7e42f9f8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-4268w\" (UID: \"1d0d721e-fbdc-4f18-88a2-094f7e42f9f8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4268w"
Oct 30 19:59:33.511019 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:33.510997    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wldw\" (UniqueName: \"kubernetes.io/projected/1d0d721e-fbdc-4f18-88a2-094f7e42f9f8-kube-api-access-8wldw\") pod \"kuadrant-operator-controller-manager-678d87766f-4268w\" (UID: \"1d0d721e-fbdc-4f18-88a2-094f7e42f9f8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4268w"
Oct 30 19:59:33.686981 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:33.686899    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4268w"
Oct 30 19:59:33.974886 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:33.974859    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4268w"]
Oct 30 19:59:33.977639 ip-10-0-137-84 kubenswrapper[5043]: W1030 19:59:33.977611    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d0d721e_fbdc_4f18_88a2_094f7e42f9f8.slice/crio-ab501c9eb205a47938d006d20d221c4349d6e725fcb3751ca7dd884df40bcc03 WatchSource:0}: Error finding container ab501c9eb205a47938d006d20d221c4349d6e725fcb3751ca7dd884df40bcc03: Status 404 returned error can't find the container with id ab501c9eb205a47938d006d20d221c4349d6e725fcb3751ca7dd884df40bcc03
Oct 30 19:59:34.136768 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:34.136713    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4268w" event={"ID":"1d0d721e-fbdc-4f18-88a2-094f7e42f9f8","Type":"ContainerStarted","Data":"bb7564ab9d2adf845d9df640eadc86475b0a04fb47e46446f1084ece8593bcd7"}
Oct 30 19:59:34.136949 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:34.136802    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4268w"
Oct 30 19:59:34.136949 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:34.136826    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4268w" event={"ID":"1d0d721e-fbdc-4f18-88a2-094f7e42f9f8","Type":"ContainerStarted","Data":"ab501c9eb205a47938d006d20d221c4349d6e725fcb3751ca7dd884df40bcc03"}
Oct 30 19:59:34.160367 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:34.160203    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4268w" podStartSLOduration=1.160186674 podStartE2EDuration="1.160186674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:34.15827454 +0000 UTC m=+2593.669334506" watchObservedRunningTime="2026-10-30 19:59:34.160186674 +0000 UTC m=+2593.671246637"
Oct 30 19:59:45.143811 ip-10-0-137-84 kubenswrapper[5043]: I1030 19:59:45.143778    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4268w"
Oct 30 19:59:55.031121 ip-10-0-137-84 kubenswrapper[5043]: E1030 19:59:55.031066    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 20:00:02.347626 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:02.347546    5043 generic.go:356] "Generic (PLEG): container finished" podID="4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21" containerID="22d0aebc9d4cec4dde38bceefc8fd9319937d384bf0d59b3e503cb49a8677fbe" exitCode=0
Oct 30 20:00:02.348072 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:02.347621    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp" event={"ID":"4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21","Type":"ContainerDied","Data":"22d0aebc9d4cec4dde38bceefc8fd9319937d384bf0d59b3e503cb49a8677fbe"}
Oct 30 20:00:02.348072 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:02.348050    5043 scope.go:122] "RemoveContainer" containerID="22d0aebc9d4cec4dde38bceefc8fd9319937d384bf0d59b3e503cb49a8677fbe"
Oct 30 20:00:02.349238 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:02.349213    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 20:00:02.349312 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:02.349275    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 20:00:02.349312 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:02.349303    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 20:00:03.354872 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:03.354832    5043 generic.go:356] "Generic (PLEG): container finished" podID="cc14aeeb-67bf-438e-870e-595e528ba5c4" containerID="6057c59d389531f0c1c1307639910671ac8ff23259caeae62c679ec68c3c1a61" exitCode=0
Oct 30 20:00:03.355335 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:03.354902    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57" event={"ID":"cc14aeeb-67bf-438e-870e-595e528ba5c4","Type":"ContainerDied","Data":"6057c59d389531f0c1c1307639910671ac8ff23259caeae62c679ec68c3c1a61"}
Oct 30 20:00:03.355388 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:03.355349    5043 scope.go:122] "RemoveContainer" containerID="6057c59d389531f0c1c1307639910671ac8ff23259caeae62c679ec68c3c1a61"
Oct 30 20:00:03.370281 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:03.365406    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 20:00:03.370281 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:03.365508    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 20:00:03.370281 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:03.365555    5043 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 20:00:03.383559 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:03.383518    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp" event={"ID":"4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21","Type":"ContainerStarted","Data":"2668e1958d5e7fdabdd453338a5664ac88934dafc1acf4702a931c127ec3eacf"}
Oct 30 20:00:03.878761 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:03.876917    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp"
Oct 30 20:00:03.878761 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:03.877051    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ftjzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body=
Oct 30 20:00:03.878761 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:03.877144    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp" podUID="4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused"
Oct 30 20:00:04.401553 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:04.401514    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57" event={"ID":"cc14aeeb-67bf-438e-870e-595e528ba5c4","Type":"ContainerStarted","Data":"d016576b67172ff96ca32cef5c8bb5782e51aee285b8da106a86edfc4a999e51"}
Oct 30 20:00:04.878760 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:04.877941    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57"
Oct 30 20:00:04.878760 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:04.878058    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ftjzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body=
Oct 30 20:00:04.878760 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:04.878104    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp" podUID="4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused"
Oct 30 20:00:04.879111 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:04.878843    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jrm57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body=
Oct 30 20:00:04.879111 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:04.878887    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57" podUID="cc14aeeb-67bf-438e-870e-595e528ba5c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused"
Oct 30 20:00:04.906164 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:04.906123    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57"
Oct 30 20:00:04.982761 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:04.982281    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp"
Oct 30 20:00:05.884387 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:05.884350    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ftjzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body=
Oct 30 20:00:05.885138 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:05.885111    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp" podUID="4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused"
Oct 30 20:00:05.885663 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:05.885045    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jrm57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body=
Oct 30 20:00:05.885846 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:05.885823    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57" podUID="cc14aeeb-67bf-438e-870e-595e528ba5c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused"
Oct 30 20:00:06.874761 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:06.874359    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jrm57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body=
Oct 30 20:00:06.874761 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:06.874420    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57" podUID="cc14aeeb-67bf-438e-870e-595e528ba5c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused"
Oct 30 20:00:06.875068 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:06.874887    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ftjzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body=
Oct 30 20:00:06.875068 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:06.874923    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp" podUID="4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused"
Oct 30 20:00:07.879379 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:07.879339    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ftjzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body=
Oct 30 20:00:07.880074 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:07.880041    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp" podUID="4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused"
Oct 30 20:00:07.880635 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:07.879833    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jrm57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body=
Oct 30 20:00:07.880843 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:07.880819    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57" podUID="cc14aeeb-67bf-438e-870e-595e528ba5c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused"
Oct 30 20:00:08.881884 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:08.881844    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ftjzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body=
Oct 30 20:00:08.882506 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:08.882471    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp" podUID="4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused"
Oct 30 20:00:08.885862 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:08.885835    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jrm57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body=
Oct 30 20:00:08.886056 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:08.886033    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57" podUID="cc14aeeb-67bf-438e-870e-595e528ba5c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused"
Oct 30 20:00:09.874103 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:09.874055    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ftjzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body=
Oct 30 20:00:09.874342 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:09.874130    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp" podUID="4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused"
Oct 30 20:00:09.874493 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:09.874460    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jrm57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body=
Oct 30 20:00:09.874572 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:09.874540    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57" podUID="cc14aeeb-67bf-438e-870e-595e528ba5c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused"
Oct 30 20:00:10.876763 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:10.875776    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jrm57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body=
Oct 30 20:00:10.876763 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:10.875863    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57" podUID="cc14aeeb-67bf-438e-870e-595e528ba5c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused"
Oct 30 20:00:11.876194 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:11.876156    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ftjzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:00:11.876500 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:11.876466    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp" podUID="4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:00:12.875323 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:12.875277    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jrm57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:00:12.875892 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:12.875375    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57" podUID="cc14aeeb-67bf-438e-870e-595e528ba5c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:00:12.917947 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:12.917907    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ftjzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:00:12.918158 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:12.917968    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp" podUID="4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:00:13.916973 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:13.916930    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jrm57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:00:13.917543 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:13.917011    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57" podUID="cc14aeeb-67bf-438e-870e-595e528ba5c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:00:13.918897 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:13.918810    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ftjzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:00:13.919121 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:13.919092    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp" podUID="4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:00:14.918042 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:14.918002    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jrm57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:00:14.918651 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:14.918611    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57" podUID="cc14aeeb-67bf-438e-870e-595e528ba5c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:00:14.960959 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:14.960905    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ftjzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:00:14.961277 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:14.961235    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp" podUID="4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:00:15.959957 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:15.959913    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jrm57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:00:15.960498 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:15.959979    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57" podUID="cc14aeeb-67bf-438e-870e-595e528ba5c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:00:15.974260 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:15.974186    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ftjzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:00:15.974425 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:15.974284    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp" podUID="4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:00:15.987861 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:15.987836    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp"
Oct 30 20:00:15.989171 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:15.989149    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp"
Oct 30 20:00:17.001907 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:17.001866    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jrm57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:00:17.002427 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:17.001936    5043 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57" podUID="cc14aeeb-67bf-438e-870e-595e528ba5c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:00:17.087455 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:17.087428    5043 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57"
Oct 30 20:00:17.088500 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:00:17.088480    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jrm57"
Oct 30 20:00:24.976157 ip-10-0-137-84 kubenswrapper[5043]: E1030 20:00:24.976115    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 20:00:54.978394 ip-10-0-137-84 kubenswrapper[5043]: E1030 20:00:54.978351    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 20:01:19.617634 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:19.617600    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-043b7b66-77jvz"]
Oct 30 20:01:19.621236 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:19.621218    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-043b7b66-77jvz"
Oct 30 20:01:19.627400 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:19.627352    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:01:19.627400 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:19.627368    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:01:19.627400 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:19.627375    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:01:19.627705 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:19.627440    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 20:01:19.630071 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:19.630049    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-043b7b66-77jvz"]
Oct 30 20:01:19.665263 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:19.665223    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5bac402-7c8a-4cbf-a511-cdf3049d3475-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-043b7b66-77jvz\" (UID: \"b5bac402-7c8a-4cbf-a511-cdf3049d3475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-043b7b66-77jvz"
Oct 30 20:01:19.665497 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:19.665273    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5bac402-7c8a-4cbf-a511-cdf3049d3475-maas-api-xks-ca\") pod \"maas-api-revoke-sub-043b7b66-77jvz\" (UID: \"b5bac402-7c8a-4cbf-a511-cdf3049d3475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-043b7b66-77jvz"
Oct 30 20:01:19.665497 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:19.665370    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xmb9\" (UniqueName: \"kubernetes.io/projected/b5bac402-7c8a-4cbf-a511-cdf3049d3475-kube-api-access-4xmb9\") pod \"maas-api-revoke-sub-043b7b66-77jvz\" (UID: \"b5bac402-7c8a-4cbf-a511-cdf3049d3475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-043b7b66-77jvz"
Oct 30 20:01:19.766258 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:19.766222    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5bac402-7c8a-4cbf-a511-cdf3049d3475-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-043b7b66-77jvz\" (UID: \"b5bac402-7c8a-4cbf-a511-cdf3049d3475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-043b7b66-77jvz"
Oct 30 20:01:19.766456 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:19.766272    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5bac402-7c8a-4cbf-a511-cdf3049d3475-maas-api-xks-ca\") pod \"maas-api-revoke-sub-043b7b66-77jvz\" (UID: \"b5bac402-7c8a-4cbf-a511-cdf3049d3475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-043b7b66-77jvz"
Oct 30 20:01:19.766456 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:19.766333    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xmb9\" (UniqueName: \"kubernetes.io/projected/b5bac402-7c8a-4cbf-a511-cdf3049d3475-kube-api-access-4xmb9\") pod \"maas-api-revoke-sub-043b7b66-77jvz\" (UID: \"b5bac402-7c8a-4cbf-a511-cdf3049d3475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-043b7b66-77jvz"
Oct 30 20:01:19.766952 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:19.766907    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5bac402-7c8a-4cbf-a511-cdf3049d3475-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-043b7b66-77jvz\" (UID: \"b5bac402-7c8a-4cbf-a511-cdf3049d3475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-043b7b66-77jvz"
Oct 30 20:01:19.769008 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:19.768981    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5bac402-7c8a-4cbf-a511-cdf3049d3475-maas-api-xks-ca\") pod \"maas-api-revoke-sub-043b7b66-77jvz\" (UID: \"b5bac402-7c8a-4cbf-a511-cdf3049d3475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-043b7b66-77jvz"
Oct 30 20:01:19.774954 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:19.774928    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xmb9\" (UniqueName: \"kubernetes.io/projected/b5bac402-7c8a-4cbf-a511-cdf3049d3475-kube-api-access-4xmb9\") pod \"maas-api-revoke-sub-043b7b66-77jvz\" (UID: \"b5bac402-7c8a-4cbf-a511-cdf3049d3475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-043b7b66-77jvz"
Oct 30 20:01:19.937332 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:19.937240    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-043b7b66-77jvz"
Oct 30 20:01:20.215244 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:20.215219    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-043b7b66-77jvz"]
Oct 30 20:01:20.217804 ip-10-0-137-84 kubenswrapper[5043]: W1030 20:01:20.217726    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5bac402_7c8a_4cbf_a511_cdf3049d3475.slice/crio-36273b18eee449f0099860d7166c30b027cbff5080339dc550f90ad86212fdc2 WatchSource:0}: Error finding container 36273b18eee449f0099860d7166c30b027cbff5080339dc550f90ad86212fdc2: Status 404 returned error can't find the container with id 36273b18eee449f0099860d7166c30b027cbff5080339dc550f90ad86212fdc2
Oct 30 20:01:20.849313 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:20.849211    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-043b7b66-77jvz" event={"ID":"b5bac402-7c8a-4cbf-a511-cdf3049d3475","Type":"ContainerStarted","Data":"55cf5dbafe8ed773eb8c62a4b9038ef40bb0b244210660c05f618586caaa87e5"}
Oct 30 20:01:20.849313 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:20.849280    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-043b7b66-77jvz" event={"ID":"b5bac402-7c8a-4cbf-a511-cdf3049d3475","Type":"ContainerStarted","Data":"36273b18eee449f0099860d7166c30b027cbff5080339dc550f90ad86212fdc2"}
Oct 30 20:01:20.895374 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:20.895313    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-043b7b66-77jvz" podStartSLOduration=1.895296266 podStartE2EDuration="1.895296266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:20.894547569 +0000 UTC m=+2700.405607533" watchObservedRunningTime="2026-10-30 20:01:20.895296266 +0000 UTC m=+2700.406356231"
Oct 30 20:01:21.287617 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:21.287583    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cmw6w_76e21916-0744-4499-a0cb-64a7a44a8285/console-operator/2.log"
Oct 30 20:01:21.292925 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:21.292895    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cmw6w_76e21916-0744-4499-a0cb-64a7a44a8285/console-operator/2.log"
Oct 30 20:01:21.856538 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:21.856497    5043 generic.go:356] "Generic (PLEG): container finished" podID="b5bac402-7c8a-4cbf-a511-cdf3049d3475" containerID="55cf5dbafe8ed773eb8c62a4b9038ef40bb0b244210660c05f618586caaa87e5" exitCode=0
Oct 30 20:01:21.856990 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:21.856588    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-043b7b66-77jvz" event={"ID":"b5bac402-7c8a-4cbf-a511-cdf3049d3475","Type":"ContainerDied","Data":"55cf5dbafe8ed773eb8c62a4b9038ef40bb0b244210660c05f618586caaa87e5"}
Oct 30 20:01:23.025817 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:23.025789    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-043b7b66-77jvz"
Oct 30 20:01:23.106156 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:23.106125    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5bac402-7c8a-4cbf-a511-cdf3049d3475-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5bac402-7c8a-4cbf-a511-cdf3049d3475-maas-api-xks-ca\") pod \"b5bac402-7c8a-4cbf-a511-cdf3049d3475\" (UID: \"b5bac402-7c8a-4cbf-a511-cdf3049d3475\") "
Oct 30 20:01:23.106361 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:23.106178    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5bac402-7c8a-4cbf-a511-cdf3049d3475-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5bac402-7c8a-4cbf-a511-cdf3049d3475-maas-api-openshift-ca\") pod \"b5bac402-7c8a-4cbf-a511-cdf3049d3475\" (UID: \"b5bac402-7c8a-4cbf-a511-cdf3049d3475\") "
Oct 30 20:01:23.106361 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:23.106268    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5bac402-7c8a-4cbf-a511-cdf3049d3475-kube-api-access-4xmb9\" (UniqueName: \"kubernetes.io/projected/b5bac402-7c8a-4cbf-a511-cdf3049d3475-kube-api-access-4xmb9\") pod \"b5bac402-7c8a-4cbf-a511-cdf3049d3475\" (UID: \"b5bac402-7c8a-4cbf-a511-cdf3049d3475\") "
Oct 30 20:01:23.106584 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:23.106556    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5bac402-7c8a-4cbf-a511-cdf3049d3475-maas-api-openshift-ca" pod "b5bac402-7c8a-4cbf-a511-cdf3049d3475" (UID: "b5bac402-7c8a-4cbf-a511-cdf3049d3475"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:23.108455 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:23.108424    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5bac402-7c8a-4cbf-a511-cdf3049d3475-maas-api-xks-ca" pod "b5bac402-7c8a-4cbf-a511-cdf3049d3475" (UID: "b5bac402-7c8a-4cbf-a511-cdf3049d3475"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:01:23.108455 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:23.108444    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5bac402-7c8a-4cbf-a511-cdf3049d3475-kube-api-access-4xmb9" pod "b5bac402-7c8a-4cbf-a511-cdf3049d3475" (UID: "b5bac402-7c8a-4cbf-a511-cdf3049d3475"). InnerVolumeSpecName "kube-api-access-4xmb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:23.207762 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:23.207656    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xmb9\" (UniqueName: \"kubernetes.io/projected/b5bac402-7c8a-4cbf-a511-cdf3049d3475-kube-api-access-4xmb9\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:23.207762 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:23.207699    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5bac402-7c8a-4cbf-a511-cdf3049d3475-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:23.207762 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:23.207714    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5bac402-7c8a-4cbf-a511-cdf3049d3475-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:23.872997 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:23.872964    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-043b7b66-77jvz"
Oct 30 20:01:23.872997 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:23.872985    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-043b7b66-77jvz" event={"ID":"b5bac402-7c8a-4cbf-a511-cdf3049d3475","Type":"ContainerDied","Data":"36273b18eee449f0099860d7166c30b027cbff5080339dc550f90ad86212fdc2"}
Oct 30 20:01:23.873223 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:23.873034    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36273b18eee449f0099860d7166c30b027cbff5080339dc550f90ad86212fdc2"
Oct 30 20:01:24.976348 ip-10-0-137-84 kubenswrapper[5043]: E1030 20:01:24.976308    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 20:01:30.631405 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:30.631364    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-cb7041-x9qzv"]
Oct 30 20:01:30.631914 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:30.631896    5043 cpu_manager.go:414] "containerMap: removing container" podUID="b5bac402-7c8a-4cbf-a511-cdf3049d3475" containerName="revoke-keys"
Oct 30 20:01:30.631966 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:30.631915    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5bac402-7c8a-4cbf-a511-cdf3049d3475" containerName="revoke-keys"
Oct 30 20:01:30.632016 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:30.632006    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5bac402-7c8a-4cbf-a511-cdf3049d3475" containerName="revoke-keys"
Oct 30 20:01:30.635513 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:30.635496    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-cb7041-x9qzv"
Oct 30 20:01:30.641970 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:30.641946    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:01:30.642156 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:30.642137    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:01:30.642241 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:30.642220    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:01:30.642241 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:30.642064    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 20:01:30.646096 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:30.646075    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-cb7041-x9qzv"]
Oct 30 20:01:30.673225 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:30.673186    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhcwr\" (UniqueName: \"kubernetes.io/projected/a2dd020f-a4c6-42e5-abf4-248fc045bd21-kube-api-access-zhcwr\") pod \"maas-api-revoke-keys-e2e-worker-single-main-cb7041-x9qzv\" (UID: \"a2dd020f-a4c6-42e5-abf4-248fc045bd21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-cb7041-x9qzv"
Oct 30 20:01:30.673419 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:30.673244    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2dd020f-a4c6-42e5-abf4-248fc045bd21-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-cb7041-x9qzv\" (UID: \"a2dd020f-a4c6-42e5-abf4-248fc045bd21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-cb7041-x9qzv"
Oct 30 20:01:30.673419 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:30.673405    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2dd020f-a4c6-42e5-abf4-248fc045bd21-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-cb7041-x9qzv\" (UID: \"a2dd020f-a4c6-42e5-abf4-248fc045bd21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-cb7041-x9qzv"
Oct 30 20:01:30.774089 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:30.774043    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2dd020f-a4c6-42e5-abf4-248fc045bd21-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-cb7041-x9qzv\" (UID: \"a2dd020f-a4c6-42e5-abf4-248fc045bd21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-cb7041-x9qzv"
Oct 30 20:01:30.774325 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:30.774188    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhcwr\" (UniqueName: \"kubernetes.io/projected/a2dd020f-a4c6-42e5-abf4-248fc045bd21-kube-api-access-zhcwr\") pod \"maas-api-revoke-keys-e2e-worker-single-main-cb7041-x9qzv\" (UID: \"a2dd020f-a4c6-42e5-abf4-248fc045bd21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-cb7041-x9qzv"
Oct 30 20:01:30.774325 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:30.774228    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2dd020f-a4c6-42e5-abf4-248fc045bd21-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-cb7041-x9qzv\" (UID: \"a2dd020f-a4c6-42e5-abf4-248fc045bd21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-cb7041-x9qzv"
Oct 30 20:01:30.774922 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:30.774884    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2dd020f-a4c6-42e5-abf4-248fc045bd21-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-cb7041-x9qzv\" (UID: \"a2dd020f-a4c6-42e5-abf4-248fc045bd21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-cb7041-x9qzv"
Oct 30 20:01:30.778153 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:30.778126    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2dd020f-a4c6-42e5-abf4-248fc045bd21-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-cb7041-x9qzv\" (UID: \"a2dd020f-a4c6-42e5-abf4-248fc045bd21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-cb7041-x9qzv"
Oct 30 20:01:30.782512 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:30.782480    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhcwr\" (UniqueName: \"kubernetes.io/projected/a2dd020f-a4c6-42e5-abf4-248fc045bd21-kube-api-access-zhcwr\") pod \"maas-api-revoke-keys-e2e-worker-single-main-cb7041-x9qzv\" (UID: \"a2dd020f-a4c6-42e5-abf4-248fc045bd21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-cb7041-x9qzv"
Oct 30 20:01:30.950877 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:30.950794    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-cb7041-x9qzv"
Oct 30 20:01:31.286962 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:31.286924    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-cb7041-x9qzv"]
Oct 30 20:01:31.290373 ip-10-0-137-84 kubenswrapper[5043]: W1030 20:01:31.290340    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2dd020f_a4c6_42e5_abf4_248fc045bd21.slice/crio-204b5a64dfa96f8e6472e4de1402d5200cbfae34320472b6042e1d309768c93a WatchSource:0}: Error finding container 204b5a64dfa96f8e6472e4de1402d5200cbfae34320472b6042e1d309768c93a: Status 404 returned error can't find the container with id 204b5a64dfa96f8e6472e4de1402d5200cbfae34320472b6042e1d309768c93a
Oct 30 20:01:31.921018 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:31.919432    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-cb7041-x9qzv" event={"ID":"a2dd020f-a4c6-42e5-abf4-248fc045bd21","Type":"ContainerStarted","Data":"da24ea850abc368fbeddce9b985922f40ef5907f93989ebe850f4376bb1a0276"}
Oct 30 20:01:31.921018 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:31.919508    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-cb7041-x9qzv" event={"ID":"a2dd020f-a4c6-42e5-abf4-248fc045bd21","Type":"ContainerStarted","Data":"204b5a64dfa96f8e6472e4de1402d5200cbfae34320472b6042e1d309768c93a"}
Oct 30 20:01:31.939705 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:31.939653    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-cb7041-x9qzv" podStartSLOduration=1.939639932 podStartE2EDuration="1.939639932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:31.93637499 +0000 UTC m=+2711.447434957" watchObservedRunningTime="2026-10-30 20:01:31.939639932 +0000 UTC m=+2711.450699914"
Oct 30 20:01:32.927720 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:32.927683    5043 generic.go:356] "Generic (PLEG): container finished" podID="a2dd020f-a4c6-42e5-abf4-248fc045bd21" containerID="da24ea850abc368fbeddce9b985922f40ef5907f93989ebe850f4376bb1a0276" exitCode=0
Oct 30 20:01:32.928134 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:32.927772    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-cb7041-x9qzv" event={"ID":"a2dd020f-a4c6-42e5-abf4-248fc045bd21","Type":"ContainerDied","Data":"da24ea850abc368fbeddce9b985922f40ef5907f93989ebe850f4376bb1a0276"}
Oct 30 20:01:34.113390 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:34.113360    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-cb7041-x9qzv"
Oct 30 20:01:34.218963 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:34.218934    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2dd020f-a4c6-42e5-abf4-248fc045bd21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2dd020f-a4c6-42e5-abf4-248fc045bd21-maas-api-openshift-ca\") pod \"a2dd020f-a4c6-42e5-abf4-248fc045bd21\" (UID: \"a2dd020f-a4c6-42e5-abf4-248fc045bd21\") "
Oct 30 20:01:34.219139 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:34.219050    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2dd020f-a4c6-42e5-abf4-248fc045bd21-kube-api-access-zhcwr\" (UniqueName: \"kubernetes.io/projected/a2dd020f-a4c6-42e5-abf4-248fc045bd21-kube-api-access-zhcwr\") pod \"a2dd020f-a4c6-42e5-abf4-248fc045bd21\" (UID: \"a2dd020f-a4c6-42e5-abf4-248fc045bd21\") "
Oct 30 20:01:34.219139 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:34.219069    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2dd020f-a4c6-42e5-abf4-248fc045bd21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2dd020f-a4c6-42e5-abf4-248fc045bd21-maas-api-xks-ca\") pod \"a2dd020f-a4c6-42e5-abf4-248fc045bd21\" (UID: \"a2dd020f-a4c6-42e5-abf4-248fc045bd21\") "
Oct 30 20:01:34.219386 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:34.219290    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2dd020f-a4c6-42e5-abf4-248fc045bd21-maas-api-openshift-ca" pod "a2dd020f-a4c6-42e5-abf4-248fc045bd21" (UID: "a2dd020f-a4c6-42e5-abf4-248fc045bd21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:34.221377 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:34.221349    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2dd020f-a4c6-42e5-abf4-248fc045bd21-kube-api-access-zhcwr" pod "a2dd020f-a4c6-42e5-abf4-248fc045bd21" (UID: "a2dd020f-a4c6-42e5-abf4-248fc045bd21"). InnerVolumeSpecName "kube-api-access-zhcwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:34.221483 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:34.221416    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2dd020f-a4c6-42e5-abf4-248fc045bd21-maas-api-xks-ca" pod "a2dd020f-a4c6-42e5-abf4-248fc045bd21" (UID: "a2dd020f-a4c6-42e5-abf4-248fc045bd21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:01:34.319965 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:34.319930    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhcwr\" (UniqueName: \"kubernetes.io/projected/a2dd020f-a4c6-42e5-abf4-248fc045bd21-kube-api-access-zhcwr\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:34.319965 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:34.319960    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2dd020f-a4c6-42e5-abf4-248fc045bd21-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:34.319965 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:34.319971    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2dd020f-a4c6-42e5-abf4-248fc045bd21-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:01:34.939903 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:34.939817    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-cb7041-x9qzv"
Oct 30 20:01:34.939903 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:34.939862    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-cb7041-x9qzv" event={"ID":"a2dd020f-a4c6-42e5-abf4-248fc045bd21","Type":"ContainerDied","Data":"204b5a64dfa96f8e6472e4de1402d5200cbfae34320472b6042e1d309768c93a"}
Oct 30 20:01:34.940103 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:34.939920    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="204b5a64dfa96f8e6472e4de1402d5200cbfae34320472b6042e1d309768c93a"
Oct 30 20:01:41.917109 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:41.917073    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-cb7041-x9qzv"]
Oct 30 20:01:41.920668 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:41.920638    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-cb7041-x9qzv"]
Oct 30 20:01:42.652454 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:01:42.652414    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2dd020f-a4c6-42e5-abf4-248fc045bd21" path="/var/lib/kubelet/pods/a2dd020f-a4c6-42e5-abf4-248fc045bd21/volumes"
Oct 30 20:01:54.977092 ip-10-0-137-84 kubenswrapper[5043]: E1030 20:01:54.977059    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 20:02:24.976366 ip-10-0-137-84 kubenswrapper[5043]: E1030 20:02:24.976284    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 20:02:40.039765 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:02:40.039245    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9621d482-j9r44"]
Oct 30 20:02:40.057493 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:02:40.057450    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9621d482-j9r44"]
Oct 30 20:02:40.648794 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:02:40.648756    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae4a2f97-73b4-4c43-ae00-c47151d4c99a" path="/var/lib/kubelet/pods/ae4a2f97-73b4-4c43-ae00-c47151d4c99a/volumes"
Oct 30 20:02:42.634882 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:02:42.634841    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-zgd28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:02:42.635313 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:02:42.634909    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zgd28" podUID="8a958e54-98cc-4651-8c1b-dcb6b6207d1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:02:47.036921 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:02:47.036856    5043 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ftjzp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:02:47.037750 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:02:47.037677    5043 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ftjzp" podUID="4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:02:54.979028 ip-10-0-137-84 kubenswrapper[5043]: E1030 20:02:54.978981    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 20:03:24.977289 ip-10-0-137-84 kubenswrapper[5043]: E1030 20:03:24.977244    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 20:03:40.987437 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:03:40.987392    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-58847295-bvjf7"]
Oct 30 20:03:40.988106 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:03:40.988084    5043 cpu_manager.go:414] "containerMap: removing container" podUID="a2dd020f-a4c6-42e5-abf4-248fc045bd21" containerName="revoke-keys"
Oct 30 20:03:40.988199 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:03:40.988108    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2dd020f-a4c6-42e5-abf4-248fc045bd21" containerName="revoke-keys"
Oct 30 20:03:40.988266 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:03:40.988233    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2dd020f-a4c6-42e5-abf4-248fc045bd21" containerName="revoke-keys"
Oct 30 20:03:40.991637 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:03:40.991617    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-58847295-bvjf7"
Oct 30 20:03:40.997461 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:03:40.997438    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\""
Oct 30 20:03:40.997592 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:03:40.997494    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:03:40.997674 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:03:40.997660    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:03:40.997768 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:03:40.997703    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:03:41.000306 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:03:41.000280    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-58847295-bvjf7"]
Oct 30 20:03:41.129692 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:03:41.129653    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3993f37-0c68-47fe-b49b-2abc80fdf248-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-58847295-bvjf7\" (UID: \"e3993f37-0c68-47fe-b49b-2abc80fdf248\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-58847295-bvjf7"
Oct 30 20:03:41.129892 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:03:41.129803    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5czg\" (UniqueName: \"kubernetes.io/projected/e3993f37-0c68-47fe-b49b-2abc80fdf248-kube-api-access-h5czg\") pod \"maas-api-revoke-keys-e2e-ait-58847295-bvjf7\" (UID: \"e3993f37-0c68-47fe-b49b-2abc80fdf248\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-58847295-bvjf7"
Oct 30 20:03:41.129892 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:03:41.129866    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3993f37-0c68-47fe-b49b-2abc80fdf248-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-58847295-bvjf7\" (UID: \"e3993f37-0c68-47fe-b49b-2abc80fdf248\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-58847295-bvjf7"
Oct 30 20:03:41.231029 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:03:41.230990    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3993f37-0c68-47fe-b49b-2abc80fdf248-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-58847295-bvjf7\" (UID: \"e3993f37-0c68-47fe-b49b-2abc80fdf248\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-58847295-bvjf7"
Oct 30 20:03:41.231214 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:03:41.231076    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5czg\" (UniqueName: \"kubernetes.io/projected/e3993f37-0c68-47fe-b49b-2abc80fdf248-kube-api-access-h5czg\") pod \"maas-api-revoke-keys-e2e-ait-58847295-bvjf7\" (UID: \"e3993f37-0c68-47fe-b49b-2abc80fdf248\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-58847295-bvjf7"
Oct 30 20:03:41.231214 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:03:41.231150    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3993f37-0c68-47fe-b49b-2abc80fdf248-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-58847295-bvjf7\" (UID: \"e3993f37-0c68-47fe-b49b-2abc80fdf248\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-58847295-bvjf7"
Oct 30 20:03:41.231719 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:03:41.231698    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3993f37-0c68-47fe-b49b-2abc80fdf248-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-58847295-bvjf7\" (UID: \"e3993f37-0c68-47fe-b49b-2abc80fdf248\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-58847295-bvjf7"
Oct 30 20:03:41.233714 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:03:41.233691    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3993f37-0c68-47fe-b49b-2abc80fdf248-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-58847295-bvjf7\" (UID: \"e3993f37-0c68-47fe-b49b-2abc80fdf248\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-58847295-bvjf7"
Oct 30 20:03:41.236331 ip-10-0-137-84 kubenswrapper[5043]: E1030 20:03:41.236310    5043 projected.go:196] Error preparing data for projected volume kube-api-access-h5czg for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-58847295-bvjf7: failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Oct 30 20:03:41.236439 ip-10-0-137-84 kubenswrapper[5043]: E1030 20:03:41.236376    5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3993f37-0c68-47fe-b49b-2abc80fdf248-kube-api-access-h5czg podName:e3993f37-0c68-47fe-b49b-2abc80fdf248 nodeName:}" failed. No retries permitted until 2026-10-30 20:03:41.736361668 +0000 UTC m=+2841.247421610 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-h5czg" (UniqueName: "kubernetes.io/projected/e3993f37-0c68-47fe-b49b-2abc80fdf248-kube-api-access-h5czg") pod "maas-api-revoke-keys-e2e-ait-58847295-bvjf7" (UID: "e3993f37-0c68-47fe-b49b-2abc80fdf248") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Oct 30 20:03:41.836647 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:03:41.836614    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5czg\" (UniqueName: \"kubernetes.io/projected/e3993f37-0c68-47fe-b49b-2abc80fdf248-kube-api-access-h5czg\") pod \"maas-api-revoke-keys-e2e-ait-58847295-bvjf7\" (UID: \"e3993f37-0c68-47fe-b49b-2abc80fdf248\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-58847295-bvjf7"
Oct 30 20:03:41.858113 ip-10-0-137-84 kubenswrapper[5043]: E1030 20:03:41.858089    5043 projected.go:196] Error preparing data for projected volume kube-api-access-h5czg for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-58847295-bvjf7: failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Oct 30 20:03:41.858212 ip-10-0-137-84 kubenswrapper[5043]: E1030 20:03:41.858151    5043 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3993f37-0c68-47fe-b49b-2abc80fdf248-kube-api-access-h5czg podName:e3993f37-0c68-47fe-b49b-2abc80fdf248 nodeName:}" failed. No retries permitted until 2026-10-30 20:03:42.8581361 +0000 UTC m=+2842.369196042 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-h5czg" (UniqueName: "kubernetes.io/projected/e3993f37-0c68-47fe-b49b-2abc80fdf248-kube-api-access-h5czg") pod "maas-api-revoke-keys-e2e-ait-58847295-bvjf7" (UID: "e3993f37-0c68-47fe-b49b-2abc80fdf248") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Oct 30 20:03:42.949068 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:03:42.949028    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5czg\" (UniqueName: \"kubernetes.io/projected/e3993f37-0c68-47fe-b49b-2abc80fdf248-kube-api-access-h5czg\") pod \"maas-api-revoke-keys-e2e-ait-58847295-bvjf7\" (UID: \"e3993f37-0c68-47fe-b49b-2abc80fdf248\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-58847295-bvjf7"
Oct 30 20:03:42.957920 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:03:42.957892    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5czg\" (UniqueName: \"kubernetes.io/projected/e3993f37-0c68-47fe-b49b-2abc80fdf248-kube-api-access-h5czg\") pod \"maas-api-revoke-keys-e2e-ait-58847295-bvjf7\" (UID: \"e3993f37-0c68-47fe-b49b-2abc80fdf248\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-58847295-bvjf7"
Oct 30 20:03:43.106401 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:03:43.106367    5043 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-58847295-bvjf7" secret="" err="secret \"maas-api-cleanup-dockercfg-vvmk9\" not found"
Oct 30 20:03:43.106598 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:03:43.106435    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-58847295-bvjf7"
Oct 30 20:03:43.381356 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:03:43.381324    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-58847295-bvjf7"]
Oct 30 20:03:43.383850 ip-10-0-137-84 kubenswrapper[5043]: W1030 20:03:43.383818    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3993f37_0c68_47fe_b49b_2abc80fdf248.slice/crio-52b9a70ff27701cb1ab0d07648bb784f10fe4d13d6fc665d238e9eb967f70e25 WatchSource:0}: Error finding container 52b9a70ff27701cb1ab0d07648bb784f10fe4d13d6fc665d238e9eb967f70e25: Status 404 returned error can't find the container with id 52b9a70ff27701cb1ab0d07648bb784f10fe4d13d6fc665d238e9eb967f70e25
Oct 30 20:03:43.847092 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:03:43.847048    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-58847295-bvjf7" event={"ID":"e3993f37-0c68-47fe-b49b-2abc80fdf248","Type":"ContainerStarted","Data":"3ef4fa4ca696a30f915a6929e0b1f641737f1a21d48af1207dd20470497a704b"}
Oct 30 20:03:43.847280 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:03:43.847112    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-58847295-bvjf7" event={"ID":"e3993f37-0c68-47fe-b49b-2abc80fdf248","Type":"ContainerStarted","Data":"52b9a70ff27701cb1ab0d07648bb784f10fe4d13d6fc665d238e9eb967f70e25"}
Oct 30 20:03:43.847341 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:03:43.847308    5043 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-58847295-bvjf7" secret="" err="secret \"maas-api-cleanup-dockercfg-vvmk9\" not found"
Oct 30 20:03:43.864131 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:03:43.864071    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-58847295-bvjf7" podStartSLOduration=3.864058122 podStartE2EDuration="3.864058122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:43.862867967 +0000 UTC m=+2843.373927945" watchObservedRunningTime="2026-10-30 20:03:43.864058122 +0000 UTC m=+2843.375118096"
Oct 30 20:03:44.854011 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:03:44.853977    5043 generic.go:356] "Generic (PLEG): container finished" podID="e3993f37-0c68-47fe-b49b-2abc80fdf248" containerID="3ef4fa4ca696a30f915a6929e0b1f641737f1a21d48af1207dd20470497a704b" exitCode=0
Oct 30 20:03:44.854434 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:03:44.854066    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-58847295-bvjf7" event={"ID":"e3993f37-0c68-47fe-b49b-2abc80fdf248","Type":"ContainerDied","Data":"3ef4fa4ca696a30f915a6929e0b1f641737f1a21d48af1207dd20470497a704b"}
Oct 30 20:03:46.018751 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:03:46.018713    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-58847295-bvjf7"
Oct 30 20:03:46.183290 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:03:46.183194    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3993f37-0c68-47fe-b49b-2abc80fdf248-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3993f37-0c68-47fe-b49b-2abc80fdf248-maas-api-xks-ca\") pod \"e3993f37-0c68-47fe-b49b-2abc80fdf248\" (UID: \"e3993f37-0c68-47fe-b49b-2abc80fdf248\") "
Oct 30 20:03:46.183290 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:03:46.183230    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3993f37-0c68-47fe-b49b-2abc80fdf248-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3993f37-0c68-47fe-b49b-2abc80fdf248-maas-api-openshift-ca\") pod \"e3993f37-0c68-47fe-b49b-2abc80fdf248\" (UID: \"e3993f37-0c68-47fe-b49b-2abc80fdf248\") "
Oct 30 20:03:46.183544 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:03:46.183353    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3993f37-0c68-47fe-b49b-2abc80fdf248-kube-api-access-h5czg\" (UniqueName: \"kubernetes.io/projected/e3993f37-0c68-47fe-b49b-2abc80fdf248-kube-api-access-h5czg\") pod \"e3993f37-0c68-47fe-b49b-2abc80fdf248\" (UID: \"e3993f37-0c68-47fe-b49b-2abc80fdf248\") "
Oct 30 20:03:46.183712 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:03:46.183688    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3993f37-0c68-47fe-b49b-2abc80fdf248-maas-api-openshift-ca" pod "e3993f37-0c68-47fe-b49b-2abc80fdf248" (UID: "e3993f37-0c68-47fe-b49b-2abc80fdf248"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:03:46.185942 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:03:46.185914    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3993f37-0c68-47fe-b49b-2abc80fdf248-kube-api-access-h5czg" pod "e3993f37-0c68-47fe-b49b-2abc80fdf248" (UID: "e3993f37-0c68-47fe-b49b-2abc80fdf248"). InnerVolumeSpecName "kube-api-access-h5czg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:03:46.186045 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:03:46.185958    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3993f37-0c68-47fe-b49b-2abc80fdf248-maas-api-xks-ca" pod "e3993f37-0c68-47fe-b49b-2abc80fdf248" (UID: "e3993f37-0c68-47fe-b49b-2abc80fdf248"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:03:46.284230 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:03:46.284199    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3993f37-0c68-47fe-b49b-2abc80fdf248-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:46.284230 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:03:46.284229    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3993f37-0c68-47fe-b49b-2abc80fdf248-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:46.284230 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:03:46.284238    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5czg\" (UniqueName: \"kubernetes.io/projected/e3993f37-0c68-47fe-b49b-2abc80fdf248-kube-api-access-h5czg\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:46.874039 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:03:46.874002    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-58847295-bvjf7" event={"ID":"e3993f37-0c68-47fe-b49b-2abc80fdf248","Type":"ContainerDied","Data":"52b9a70ff27701cb1ab0d07648bb784f10fe4d13d6fc665d238e9eb967f70e25"}
Oct 30 20:03:46.874039 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:03:46.874037    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-58847295-bvjf7"
Oct 30 20:03:46.874271 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:03:46.874058    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52b9a70ff27701cb1ab0d07648bb784f10fe4d13d6fc665d238e9eb967f70e25"
Oct 30 20:03:52.375232 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:03:52.375187    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-58847295-bvjf7"]
Oct 30 20:03:52.378219 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:03:52.378194    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-58847295-bvjf7"]
Oct 30 20:03:52.645417 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:03:52.645318    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3993f37-0c68-47fe-b49b-2abc80fdf248" path="/var/lib/kubelet/pods/e3993f37-0c68-47fe-b49b-2abc80fdf248/volumes"
Oct 30 20:03:54.976424 ip-10-0-137-84 kubenswrapper[5043]: E1030 20:03:54.976383    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 20:04:24.976575 ip-10-0-137-84 kubenswrapper[5043]: E1030 20:04:24.976536    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 20:04:26.136262 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:04:26.136225    5043 scope.go:122] "RemoveContainer" containerID="125f8b0dc6a5042fa13e42879e6c1784bb3cfcdfb2d00d1c152046801a80411f"
Oct 30 20:04:26.145171 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:04:26.145150    5043 scope.go:122] "RemoveContainer" containerID="1d4bf6b55754c84f31d759e94b57aa39e1e36181f55ccf235fbc3e102760939c"
Oct 30 20:04:54.977534 ip-10-0-137-84 kubenswrapper[5043]: E1030 20:04:54.977493    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 20:05:24.976250 ip-10-0-137-84 kubenswrapper[5043]: E1030 20:05:24.976213    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 20:05:37.142720 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:05:37.142683    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-64f2bb-wrfrw"]
Oct 30 20:05:37.143288 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:05:37.143164    5043 cpu_manager.go:414] "containerMap: removing container" podUID="e3993f37-0c68-47fe-b49b-2abc80fdf248" containerName="revoke-keys"
Oct 30 20:05:37.143288 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:05:37.143177    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3993f37-0c68-47fe-b49b-2abc80fdf248" containerName="revoke-keys"
Oct 30 20:05:37.143288 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:05:37.143264    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3993f37-0c68-47fe-b49b-2abc80fdf248" containerName="revoke-keys"
Oct 30 20:05:37.146342 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:05:37.146324    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-64f2bb-wrfrw"
Oct 30 20:05:37.151267 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:05:37.151246    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:05:37.151267 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:05:37.151247    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:05:37.151419 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:05:37.151389    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbhlw\""
Oct 30 20:05:37.151456 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:05:37.151421    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:05:37.158110 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:05:37.158080    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-64f2bb-wrfrw"]
Oct 30 20:05:37.297611 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:05:37.297572    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr9vn\" (UniqueName: \"kubernetes.io/projected/6338d1fc-6ae6-465a-955b-61d13cb33f96-kube-api-access-gr9vn\") pod \"maas-api-revoke-keys-e2e-authcfg-http-64f2bb-wrfrw\" (UID: \"6338d1fc-6ae6-465a-955b-61d13cb33f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-64f2bb-wrfrw"
Oct 30 20:05:37.297840 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:05:37.297722    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6338d1fc-6ae6-465a-955b-61d13cb33f96-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-64f2bb-wrfrw\" (UID: \"6338d1fc-6ae6-465a-955b-61d13cb33f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-64f2bb-wrfrw"
Oct 30 20:05:37.297840 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:05:37.297818    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6338d1fc-6ae6-465a-955b-61d13cb33f96-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-64f2bb-wrfrw\" (UID: \"6338d1fc-6ae6-465a-955b-61d13cb33f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-64f2bb-wrfrw"
Oct 30 20:05:37.398697 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:05:37.398605    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6338d1fc-6ae6-465a-955b-61d13cb33f96-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-64f2bb-wrfrw\" (UID: \"6338d1fc-6ae6-465a-955b-61d13cb33f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-64f2bb-wrfrw"
Oct 30 20:05:37.398697 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:05:37.398678    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr9vn\" (UniqueName: \"kubernetes.io/projected/6338d1fc-6ae6-465a-955b-61d13cb33f96-kube-api-access-gr9vn\") pod \"maas-api-revoke-keys-e2e-authcfg-http-64f2bb-wrfrw\" (UID: \"6338d1fc-6ae6-465a-955b-61d13cb33f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-64f2bb-wrfrw"
Oct 30 20:05:37.398947 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:05:37.398811    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6338d1fc-6ae6-465a-955b-61d13cb33f96-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-64f2bb-wrfrw\" (UID: \"6338d1fc-6ae6-465a-955b-61d13cb33f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-64f2bb-wrfrw"
Oct 30 20:05:37.399478 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:05:37.399446    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6338d1fc-6ae6-465a-955b-61d13cb33f96-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-64f2bb-wrfrw\" (UID: \"6338d1fc-6ae6-465a-955b-61d13cb33f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-64f2bb-wrfrw"
Oct 30 20:05:37.401443 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:05:37.401420    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6338d1fc-6ae6-465a-955b-61d13cb33f96-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-64f2bb-wrfrw\" (UID: \"6338d1fc-6ae6-465a-955b-61d13cb33f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-64f2bb-wrfrw"
Oct 30 20:05:37.407596 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:05:37.407575    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr9vn\" (UniqueName: \"kubernetes.io/projected/6338d1fc-6ae6-465a-955b-61d13cb33f96-kube-api-access-gr9vn\") pod \"maas-api-revoke-keys-e2e-authcfg-http-64f2bb-wrfrw\" (UID: \"6338d1fc-6ae6-465a-955b-61d13cb33f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-64f2bb-wrfrw"
Oct 30 20:05:37.461542 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:05:37.461510    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-64f2bb-wrfrw"
Oct 30 20:05:37.739396 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:05:37.739367    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-64f2bb-wrfrw"]
Oct 30 20:05:37.740890 ip-10-0-137-84 kubenswrapper[5043]: W1030 20:05:37.740861    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6338d1fc_6ae6_465a_955b_61d13cb33f96.slice/crio-c6d60da96c1d01037586a30dde14b9f68813fb3a908d82246a7a0ee41681a8ae WatchSource:0}: Error finding container c6d60da96c1d01037586a30dde14b9f68813fb3a908d82246a7a0ee41681a8ae: Status 404 returned error can't find the container with id c6d60da96c1d01037586a30dde14b9f68813fb3a908d82246a7a0ee41681a8ae
Oct 30 20:05:38.510425 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:05:38.510389    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-64f2bb-wrfrw" event={"ID":"6338d1fc-6ae6-465a-955b-61d13cb33f96","Type":"ContainerStarted","Data":"3b089c03964e5a0a242817073a6296cf5fa4b37172fb6ec2982257a11f0d587d"}
Oct 30 20:05:38.510912 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:05:38.510441    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-64f2bb-wrfrw" event={"ID":"6338d1fc-6ae6-465a-955b-61d13cb33f96","Type":"ContainerStarted","Data":"c6d60da96c1d01037586a30dde14b9f68813fb3a908d82246a7a0ee41681a8ae"}
Oct 30 20:05:38.534090 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:05:38.534040    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-64f2bb-wrfrw" podStartSLOduration=1.534026793 podStartE2EDuration="1.534026793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:38.531980386 +0000 UTC m=+2958.043040350" watchObservedRunningTime="2026-10-30 20:05:38.534026793 +0000 UTC m=+2958.045086771"
Oct 30 20:05:39.517999 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:05:39.517956    5043 generic.go:356] "Generic (PLEG): container finished" podID="6338d1fc-6ae6-465a-955b-61d13cb33f96" containerID="3b089c03964e5a0a242817073a6296cf5fa4b37172fb6ec2982257a11f0d587d" exitCode=0
Oct 30 20:05:39.518506 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:05:39.518041    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-64f2bb-wrfrw" event={"ID":"6338d1fc-6ae6-465a-955b-61d13cb33f96","Type":"ContainerDied","Data":"3b089c03964e5a0a242817073a6296cf5fa4b37172fb6ec2982257a11f0d587d"}
Oct 30 20:05:40.670528 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:05:40.670497    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-64f2bb-wrfrw"
Oct 30 20:05:40.829975 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:05:40.829871    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6338d1fc-6ae6-465a-955b-61d13cb33f96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6338d1fc-6ae6-465a-955b-61d13cb33f96-maas-api-xks-ca\") pod \"6338d1fc-6ae6-465a-955b-61d13cb33f96\" (UID: \"6338d1fc-6ae6-465a-955b-61d13cb33f96\") "
Oct 30 20:05:40.830160 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:05:40.830043    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6338d1fc-6ae6-465a-955b-61d13cb33f96-kube-api-access-gr9vn\" (UniqueName: \"kubernetes.io/projected/6338d1fc-6ae6-465a-955b-61d13cb33f96-kube-api-access-gr9vn\") pod \"6338d1fc-6ae6-465a-955b-61d13cb33f96\" (UID: \"6338d1fc-6ae6-465a-955b-61d13cb33f96\") "
Oct 30 20:05:40.830160 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:05:40.830142    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6338d1fc-6ae6-465a-955b-61d13cb33f96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6338d1fc-6ae6-465a-955b-61d13cb33f96-maas-api-openshift-ca\") pod \"6338d1fc-6ae6-465a-955b-61d13cb33f96\" (UID: \"6338d1fc-6ae6-465a-955b-61d13cb33f96\") "
Oct 30 20:05:40.830466 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:05:40.830439    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6338d1fc-6ae6-465a-955b-61d13cb33f96-maas-api-openshift-ca" pod "6338d1fc-6ae6-465a-955b-61d13cb33f96" (UID: "6338d1fc-6ae6-465a-955b-61d13cb33f96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:05:40.830594 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:05:40.830580    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6338d1fc-6ae6-465a-955b-61d13cb33f96-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:40.832364 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:05:40.832334    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6338d1fc-6ae6-465a-955b-61d13cb33f96-maas-api-xks-ca" pod "6338d1fc-6ae6-465a-955b-61d13cb33f96" (UID: "6338d1fc-6ae6-465a-955b-61d13cb33f96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:05:40.832480 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:05:40.832470    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6338d1fc-6ae6-465a-955b-61d13cb33f96-kube-api-access-gr9vn" pod "6338d1fc-6ae6-465a-955b-61d13cb33f96" (UID: "6338d1fc-6ae6-465a-955b-61d13cb33f96"). InnerVolumeSpecName "kube-api-access-gr9vn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:05:40.931357 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:05:40.931314    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6338d1fc-6ae6-465a-955b-61d13cb33f96-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:40.931357 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:05:40.931355    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr9vn\" (UniqueName: \"kubernetes.io/projected/6338d1fc-6ae6-465a-955b-61d13cb33f96-kube-api-access-gr9vn\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:41.530937 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:05:41.530905    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-64f2bb-wrfrw"
Oct 30 20:05:41.531151 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:05:41.530943    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-64f2bb-wrfrw" event={"ID":"6338d1fc-6ae6-465a-955b-61d13cb33f96","Type":"ContainerDied","Data":"c6d60da96c1d01037586a30dde14b9f68813fb3a908d82246a7a0ee41681a8ae"}
Oct 30 20:05:41.531151 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:05:41.530999    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6d60da96c1d01037586a30dde14b9f68813fb3a908d82246a7a0ee41681a8ae"
Oct 30 20:05:52.890017 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:05:52.889981    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-64f2bb-wrfrw"]
Oct 30 20:05:52.891842 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:05:52.891818    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-64f2bb-wrfrw"]
Oct 30 20:05:54.644094 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:05:54.644058    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6338d1fc-6ae6-465a-955b-61d13cb33f96" path="/var/lib/kubelet/pods/6338d1fc-6ae6-465a-955b-61d13cb33f96/volumes"
Oct 30 20:05:54.978508 ip-10-0-137-84 kubenswrapper[5043]: E1030 20:05:54.978464    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 20:06:11.648397 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:11.648363    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-fq9jj"]
Oct 30 20:06:11.648956 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:11.648802    5043 cpu_manager.go:414] "containerMap: removing container" podUID="6338d1fc-6ae6-465a-955b-61d13cb33f96" containerName="revoke-keys"
Oct 30 20:06:11.648956 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:11.648813    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6338d1fc-6ae6-465a-955b-61d13cb33f96" containerName="revoke-keys"
Oct 30 20:06:11.648956 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:11.648907    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="6338d1fc-6ae6-465a-955b-61d13cb33f96" containerName="revoke-keys"
Oct 30 20:06:11.653212 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:11.653191    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-fq9jj"
Oct 30 20:06:11.658949 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:11.658920    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:06:11.658949 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:11.658942    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:06:11.659182 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:11.659006    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:06:11.659278 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:11.659254    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbhlw\""
Oct 30 20:06:11.662055 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:11.662032    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-fq9jj"]
Oct 30 20:06:11.755611 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:11.749952    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xmlv\" (UniqueName: \"kubernetes.io/projected/cf450d63-f50f-490d-91b6-21c8cc1f0fb6-kube-api-access-4xmlv\") pod \"maas-api-revoke-sub-f03629db-fq9jj\" (UID: \"cf450d63-f50f-490d-91b6-21c8cc1f0fb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-fq9jj"
Oct 30 20:06:11.755611 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:11.750107    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf450d63-f50f-490d-91b6-21c8cc1f0fb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-fq9jj\" (UID: \"cf450d63-f50f-490d-91b6-21c8cc1f0fb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-fq9jj"
Oct 30 20:06:11.755611 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:11.750303    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf450d63-f50f-490d-91b6-21c8cc1f0fb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-fq9jj\" (UID: \"cf450d63-f50f-490d-91b6-21c8cc1f0fb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-fq9jj"
Oct 30 20:06:11.851905 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:11.851861    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf450d63-f50f-490d-91b6-21c8cc1f0fb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-fq9jj\" (UID: \"cf450d63-f50f-490d-91b6-21c8cc1f0fb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-fq9jj"
Oct 30 20:06:11.852139 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:11.851922    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xmlv\" (UniqueName: \"kubernetes.io/projected/cf450d63-f50f-490d-91b6-21c8cc1f0fb6-kube-api-access-4xmlv\") pod \"maas-api-revoke-sub-f03629db-fq9jj\" (UID: \"cf450d63-f50f-490d-91b6-21c8cc1f0fb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-fq9jj"
Oct 30 20:06:11.852139 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:11.851959    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf450d63-f50f-490d-91b6-21c8cc1f0fb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-fq9jj\" (UID: \"cf450d63-f50f-490d-91b6-21c8cc1f0fb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-fq9jj"
Oct 30 20:06:11.852612 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:11.852591    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf450d63-f50f-490d-91b6-21c8cc1f0fb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-fq9jj\" (UID: \"cf450d63-f50f-490d-91b6-21c8cc1f0fb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-fq9jj"
Oct 30 20:06:11.854638 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:11.854612    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf450d63-f50f-490d-91b6-21c8cc1f0fb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-fq9jj\" (UID: \"cf450d63-f50f-490d-91b6-21c8cc1f0fb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-fq9jj"
Oct 30 20:06:11.860957 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:11.860936    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xmlv\" (UniqueName: \"kubernetes.io/projected/cf450d63-f50f-490d-91b6-21c8cc1f0fb6-kube-api-access-4xmlv\") pod \"maas-api-revoke-sub-f03629db-fq9jj\" (UID: \"cf450d63-f50f-490d-91b6-21c8cc1f0fb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-fq9jj"
Oct 30 20:06:11.968710 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:11.968674    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-fq9jj"
Oct 30 20:06:12.248590 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:12.248558    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-fq9jj"]
Oct 30 20:06:12.250814 ip-10-0-137-84 kubenswrapper[5043]: W1030 20:06:12.250786    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf450d63_f50f_490d_91b6_21c8cc1f0fb6.slice/crio-5be48109268eb5c47da064e57ab062a92fe9818e9e0442d50f7d1c7de4a13249 WatchSource:0}: Error finding container 5be48109268eb5c47da064e57ab062a92fe9818e9e0442d50f7d1c7de4a13249: Status 404 returned error can't find the container with id 5be48109268eb5c47da064e57ab062a92fe9818e9e0442d50f7d1c7de4a13249
Oct 30 20:06:12.759870 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:12.759831    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-fq9jj" event={"ID":"cf450d63-f50f-490d-91b6-21c8cc1f0fb6","Type":"ContainerStarted","Data":"c2ed520a85becf81da40ff6536e462387b57f16f3420892e5f81268bd90ea886"}
Oct 30 20:06:12.760277 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:12.759890    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-fq9jj" event={"ID":"cf450d63-f50f-490d-91b6-21c8cc1f0fb6","Type":"ContainerStarted","Data":"5be48109268eb5c47da064e57ab062a92fe9818e9e0442d50f7d1c7de4a13249"}
Oct 30 20:06:12.777838 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:12.777788    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-fq9jj" podStartSLOduration=1.777775047 podStartE2EDuration="1.777775047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:12.776564075 +0000 UTC m=+2992.287624039" watchObservedRunningTime="2026-10-30 20:06:12.777775047 +0000 UTC m=+2992.288835011"
Oct 30 20:06:13.766659 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:13.766617    5043 generic.go:356] "Generic (PLEG): container finished" podID="cf450d63-f50f-490d-91b6-21c8cc1f0fb6" containerID="c2ed520a85becf81da40ff6536e462387b57f16f3420892e5f81268bd90ea886" exitCode=0
Oct 30 20:06:13.767152 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:13.766704    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-fq9jj" event={"ID":"cf450d63-f50f-490d-91b6-21c8cc1f0fb6","Type":"ContainerDied","Data":"c2ed520a85becf81da40ff6536e462387b57f16f3420892e5f81268bd90ea886"}
Oct 30 20:06:14.924204 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:14.924180    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-fq9jj"
Oct 30 20:06:14.983060 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:14.983031    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf450d63-f50f-490d-91b6-21c8cc1f0fb6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf450d63-f50f-490d-91b6-21c8cc1f0fb6-maas-api-openshift-ca\") pod \"cf450d63-f50f-490d-91b6-21c8cc1f0fb6\" (UID: \"cf450d63-f50f-490d-91b6-21c8cc1f0fb6\") "
Oct 30 20:06:14.983250 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:14.983093    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf450d63-f50f-490d-91b6-21c8cc1f0fb6-kube-api-access-4xmlv\" (UniqueName: \"kubernetes.io/projected/cf450d63-f50f-490d-91b6-21c8cc1f0fb6-kube-api-access-4xmlv\") pod \"cf450d63-f50f-490d-91b6-21c8cc1f0fb6\" (UID: \"cf450d63-f50f-490d-91b6-21c8cc1f0fb6\") "
Oct 30 20:06:14.983250 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:14.983206    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf450d63-f50f-490d-91b6-21c8cc1f0fb6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf450d63-f50f-490d-91b6-21c8cc1f0fb6-maas-api-xks-ca\") pod \"cf450d63-f50f-490d-91b6-21c8cc1f0fb6\" (UID: \"cf450d63-f50f-490d-91b6-21c8cc1f0fb6\") "
Oct 30 20:06:14.983470 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:14.983443    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf450d63-f50f-490d-91b6-21c8cc1f0fb6-maas-api-openshift-ca" pod "cf450d63-f50f-490d-91b6-21c8cc1f0fb6" (UID: "cf450d63-f50f-490d-91b6-21c8cc1f0fb6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:06:14.985471 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:14.985435    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf450d63-f50f-490d-91b6-21c8cc1f0fb6-maas-api-xks-ca" pod "cf450d63-f50f-490d-91b6-21c8cc1f0fb6" (UID: "cf450d63-f50f-490d-91b6-21c8cc1f0fb6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:06:14.985596 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:14.985509    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf450d63-f50f-490d-91b6-21c8cc1f0fb6-kube-api-access-4xmlv" pod "cf450d63-f50f-490d-91b6-21c8cc1f0fb6" (UID: "cf450d63-f50f-490d-91b6-21c8cc1f0fb6"). InnerVolumeSpecName "kube-api-access-4xmlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:06:15.084633 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:15.084545    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xmlv\" (UniqueName: \"kubernetes.io/projected/cf450d63-f50f-490d-91b6-21c8cc1f0fb6-kube-api-access-4xmlv\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:06:15.084633 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:15.084574    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf450d63-f50f-490d-91b6-21c8cc1f0fb6-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:06:15.084633 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:15.084589    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf450d63-f50f-490d-91b6-21c8cc1f0fb6-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:06:15.779074 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:15.779045    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-fq9jj"
Oct 30 20:06:15.779074 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:15.779054    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-fq9jj" event={"ID":"cf450d63-f50f-490d-91b6-21c8cc1f0fb6","Type":"ContainerDied","Data":"5be48109268eb5c47da064e57ab062a92fe9818e9e0442d50f7d1c7de4a13249"}
Oct 30 20:06:15.779352 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:15.779096    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5be48109268eb5c47da064e57ab062a92fe9818e9e0442d50f7d1c7de4a13249"
Oct 30 20:06:21.322519 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:21.322483    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cmw6w_76e21916-0744-4499-a0cb-64a7a44a8285/console-operator/2.log"
Oct 30 20:06:21.334280 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:21.334255    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cmw6w_76e21916-0744-4499-a0cb-64a7a44a8285/console-operator/2.log"
Oct 30 20:06:24.055784 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:24.055716    5043 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-043b7b66-77jvz"]
Oct 30 20:06:24.058968 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:24.058942    5043 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-043b7b66-77jvz"]
Oct 30 20:06:24.645617 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:24.645573    5043 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5bac402-7c8a-4cbf-a511-cdf3049d3475" path="/var/lib/kubelet/pods/b5bac402-7c8a-4cbf-a511-cdf3049d3475/volumes"
Oct 30 20:06:24.982640 ip-10-0-137-84 kubenswrapper[5043]: E1030 20:06:24.982595    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 20:06:27.512434 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:27.512395    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6dsh9"]
Oct 30 20:06:27.512885 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:27.512872    5043 cpu_manager.go:414] "containerMap: removing container" podUID="cf450d63-f50f-490d-91b6-21c8cc1f0fb6" containerName="revoke-keys"
Oct 30 20:06:27.512885 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:27.512884    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf450d63-f50f-490d-91b6-21c8cc1f0fb6" containerName="revoke-keys"
Oct 30 20:06:27.513004 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:27.512994    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf450d63-f50f-490d-91b6-21c8cc1f0fb6" containerName="revoke-keys"
Oct 30 20:06:27.516276 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:27.516259    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6dsh9"
Oct 30 20:06:27.527760 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:27.527709    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:06:27.527975 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:27.527957    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbhlw\""
Oct 30 20:06:27.542341 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:27.542319    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:06:27.542506 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:27.542494    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:06:27.556846 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:27.556811    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6dsh9"]
Oct 30 20:06:27.654610 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:27.654565    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e337bb42-030e-4061-828a-cb7a470e1564-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-6dsh9\" (UID: \"e337bb42-030e-4061-828a-cb7a470e1564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6dsh9"
Oct 30 20:06:27.654836 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:27.654624    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gssf5\" (UniqueName: \"kubernetes.io/projected/e337bb42-030e-4061-828a-cb7a470e1564-kube-api-access-gssf5\") pod \"maas-api-revoke-sub-0165d7d2-6dsh9\" (UID: \"e337bb42-030e-4061-828a-cb7a470e1564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6dsh9"
Oct 30 20:06:27.654836 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:27.654815    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e337bb42-030e-4061-828a-cb7a470e1564-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-6dsh9\" (UID: \"e337bb42-030e-4061-828a-cb7a470e1564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6dsh9"
Oct 30 20:06:27.756434 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:27.756393    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e337bb42-030e-4061-828a-cb7a470e1564-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-6dsh9\" (UID: \"e337bb42-030e-4061-828a-cb7a470e1564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6dsh9"
Oct 30 20:06:27.756648 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:27.756449    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gssf5\" (UniqueName: \"kubernetes.io/projected/e337bb42-030e-4061-828a-cb7a470e1564-kube-api-access-gssf5\") pod \"maas-api-revoke-sub-0165d7d2-6dsh9\" (UID: \"e337bb42-030e-4061-828a-cb7a470e1564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6dsh9"
Oct 30 20:06:27.756648 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:27.756543    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e337bb42-030e-4061-828a-cb7a470e1564-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-6dsh9\" (UID: \"e337bb42-030e-4061-828a-cb7a470e1564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6dsh9"
Oct 30 20:06:27.757132 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:27.757105    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e337bb42-030e-4061-828a-cb7a470e1564-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-6dsh9\" (UID: \"e337bb42-030e-4061-828a-cb7a470e1564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6dsh9"
Oct 30 20:06:27.759137 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:27.759117    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e337bb42-030e-4061-828a-cb7a470e1564-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-6dsh9\" (UID: \"e337bb42-030e-4061-828a-cb7a470e1564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6dsh9"
Oct 30 20:06:27.776759 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:27.776665    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gssf5\" (UniqueName: \"kubernetes.io/projected/e337bb42-030e-4061-828a-cb7a470e1564-kube-api-access-gssf5\") pod \"maas-api-revoke-sub-0165d7d2-6dsh9\" (UID: \"e337bb42-030e-4061-828a-cb7a470e1564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6dsh9"
Oct 30 20:06:27.829608 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:27.829575    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6dsh9"
Oct 30 20:06:28.114086 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:28.114057    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6dsh9"]
Oct 30 20:06:28.116417 ip-10-0-137-84 kubenswrapper[5043]: W1030 20:06:28.116391    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode337bb42_030e_4061_828a_cb7a470e1564.slice/crio-eb5501e68ace88a04ed36cfe16bd310a1ca9adca499662ba28b970b956438f65 WatchSource:0}: Error finding container eb5501e68ace88a04ed36cfe16bd310a1ca9adca499662ba28b970b956438f65: Status 404 returned error can't find the container with id eb5501e68ace88a04ed36cfe16bd310a1ca9adca499662ba28b970b956438f65
Oct 30 20:06:28.864118 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:28.864078    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6dsh9" event={"ID":"e337bb42-030e-4061-828a-cb7a470e1564","Type":"ContainerStarted","Data":"5c1bad440891b55b3746e85bd50052be81a404b0a5b8a9af089def5979708519"}
Oct 30 20:06:28.864521 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:28.864146    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6dsh9" event={"ID":"e337bb42-030e-4061-828a-cb7a470e1564","Type":"ContainerStarted","Data":"eb5501e68ace88a04ed36cfe16bd310a1ca9adca499662ba28b970b956438f65"}
Oct 30 20:06:28.882197 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:28.882132    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6dsh9" podStartSLOduration=1.882113995 podStartE2EDuration="1.882113995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:28.881588865 +0000 UTC m=+3008.392648823" watchObservedRunningTime="2026-10-30 20:06:28.882113995 +0000 UTC m=+3008.393173962"
Oct 30 20:06:29.871594 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:29.871560    5043 generic.go:356] "Generic (PLEG): container finished" podID="e337bb42-030e-4061-828a-cb7a470e1564" containerID="5c1bad440891b55b3746e85bd50052be81a404b0a5b8a9af089def5979708519" exitCode=0
Oct 30 20:06:29.872047 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:29.871647    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6dsh9" event={"ID":"e337bb42-030e-4061-828a-cb7a470e1564","Type":"ContainerDied","Data":"5c1bad440891b55b3746e85bd50052be81a404b0a5b8a9af089def5979708519"}
Oct 30 20:06:31.048975 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:31.048946    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6dsh9"
Oct 30 20:06:31.188876 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:31.188791    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e337bb42-030e-4061-828a-cb7a470e1564-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e337bb42-030e-4061-828a-cb7a470e1564-maas-api-xks-ca\") pod \"e337bb42-030e-4061-828a-cb7a470e1564\" (UID: \"e337bb42-030e-4061-828a-cb7a470e1564\") "
Oct 30 20:06:31.189037 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:31.188880    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e337bb42-030e-4061-828a-cb7a470e1564-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e337bb42-030e-4061-828a-cb7a470e1564-maas-api-openshift-ca\") pod \"e337bb42-030e-4061-828a-cb7a470e1564\" (UID: \"e337bb42-030e-4061-828a-cb7a470e1564\") "
Oct 30 20:06:31.189037 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:31.188929    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e337bb42-030e-4061-828a-cb7a470e1564-kube-api-access-gssf5\" (UniqueName: \"kubernetes.io/projected/e337bb42-030e-4061-828a-cb7a470e1564-kube-api-access-gssf5\") pod \"e337bb42-030e-4061-828a-cb7a470e1564\" (UID: \"e337bb42-030e-4061-828a-cb7a470e1564\") "
Oct 30 20:06:31.189294 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:31.189268    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e337bb42-030e-4061-828a-cb7a470e1564-maas-api-openshift-ca" pod "e337bb42-030e-4061-828a-cb7a470e1564" (UID: "e337bb42-030e-4061-828a-cb7a470e1564"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:06:31.191324 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:31.191291    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e337bb42-030e-4061-828a-cb7a470e1564-maas-api-xks-ca" pod "e337bb42-030e-4061-828a-cb7a470e1564" (UID: "e337bb42-030e-4061-828a-cb7a470e1564"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:06:31.191324 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:31.191310    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e337bb42-030e-4061-828a-cb7a470e1564-kube-api-access-gssf5" pod "e337bb42-030e-4061-828a-cb7a470e1564" (UID: "e337bb42-030e-4061-828a-cb7a470e1564"). InnerVolumeSpecName "kube-api-access-gssf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:06:31.290432 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:31.290398    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gssf5\" (UniqueName: \"kubernetes.io/projected/e337bb42-030e-4061-828a-cb7a470e1564-kube-api-access-gssf5\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:06:31.290432 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:31.290429    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e337bb42-030e-4061-828a-cb7a470e1564-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:06:31.290432 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:31.290440    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e337bb42-030e-4061-828a-cb7a470e1564-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:06:31.883450 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:31.883420    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6dsh9"
Oct 30 20:06:31.883637 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:31.883457    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6dsh9" event={"ID":"e337bb42-030e-4061-828a-cb7a470e1564","Type":"ContainerDied","Data":"eb5501e68ace88a04ed36cfe16bd310a1ca9adca499662ba28b970b956438f65"}
Oct 30 20:06:31.883637 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:31.883505    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb5501e68ace88a04ed36cfe16bd310a1ca9adca499662ba28b970b956438f65"
Oct 30 20:06:38.354213 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:38.354174    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-s5wgg"]
Oct 30 20:06:38.354671 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:38.354653    5043 cpu_manager.go:414] "containerMap: removing container" podUID="e337bb42-030e-4061-828a-cb7a470e1564" containerName="revoke-keys"
Oct 30 20:06:38.354717 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:38.354673    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e337bb42-030e-4061-828a-cb7a470e1564" containerName="revoke-keys"
Oct 30 20:06:38.354810 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:38.354800    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="e337bb42-030e-4061-828a-cb7a470e1564" containerName="revoke-keys"
Oct 30 20:06:38.357961 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:38.357943    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-s5wgg"
Oct 30 20:06:38.362171 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:38.362142    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:06:38.362570 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:38.362538    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98db98c6-bca0-4bbc-9fdd-12912bbd1fd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-s5wgg\" (UID: \"98db98c6-bca0-4bbc-9fdd-12912bbd1fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-s5wgg"
Oct 30 20:06:38.362726 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:38.362620    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6l7n\" (UniqueName: \"kubernetes.io/projected/98db98c6-bca0-4bbc-9fdd-12912bbd1fd9-kube-api-access-p6l7n\") pod \"maas-api-revoke-sub-6b19338f-s5wgg\" (UID: \"98db98c6-bca0-4bbc-9fdd-12912bbd1fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-s5wgg"
Oct 30 20:06:38.362726 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:38.362659    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98db98c6-bca0-4bbc-9fdd-12912bbd1fd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-s5wgg\" (UID: \"98db98c6-bca0-4bbc-9fdd-12912bbd1fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-s5wgg"
Oct 30 20:06:38.363346 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:38.363323    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:06:38.364029 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:38.364009    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jbhlw\""
Oct 30 20:06:38.364203 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:38.364189    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:06:38.365451 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:38.365427    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-s5wgg"]
Oct 30 20:06:38.463602 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:38.463559    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98db98c6-bca0-4bbc-9fdd-12912bbd1fd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-s5wgg\" (UID: \"98db98c6-bca0-4bbc-9fdd-12912bbd1fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-s5wgg"
Oct 30 20:06:38.463868 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:38.463634    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6l7n\" (UniqueName: \"kubernetes.io/projected/98db98c6-bca0-4bbc-9fdd-12912bbd1fd9-kube-api-access-p6l7n\") pod \"maas-api-revoke-sub-6b19338f-s5wgg\" (UID: \"98db98c6-bca0-4bbc-9fdd-12912bbd1fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-s5wgg"
Oct 30 20:06:38.463868 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:38.463672    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98db98c6-bca0-4bbc-9fdd-12912bbd1fd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-s5wgg\" (UID: \"98db98c6-bca0-4bbc-9fdd-12912bbd1fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-s5wgg"
Oct 30 20:06:38.464659 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:38.464631    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98db98c6-bca0-4bbc-9fdd-12912bbd1fd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-s5wgg\" (UID: \"98db98c6-bca0-4bbc-9fdd-12912bbd1fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-s5wgg"
Oct 30 20:06:38.466649 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:38.466623    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98db98c6-bca0-4bbc-9fdd-12912bbd1fd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-s5wgg\" (UID: \"98db98c6-bca0-4bbc-9fdd-12912bbd1fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-s5wgg"
Oct 30 20:06:38.472629 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:38.472595    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6l7n\" (UniqueName: \"kubernetes.io/projected/98db98c6-bca0-4bbc-9fdd-12912bbd1fd9-kube-api-access-p6l7n\") pod \"maas-api-revoke-sub-6b19338f-s5wgg\" (UID: \"98db98c6-bca0-4bbc-9fdd-12912bbd1fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-s5wgg"
Oct 30 20:06:38.677338 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:38.676947    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-s5wgg"
Oct 30 20:06:39.001888 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:39.001851    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-s5wgg"]
Oct 30 20:06:39.003540 ip-10-0-137-84 kubenswrapper[5043]: W1030 20:06:39.003500    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98db98c6_bca0_4bbc_9fdd_12912bbd1fd9.slice/crio-24a1eec92d526113a346967bb4726720d1887f1793e47fbdafd4f6a41f5e2f52 WatchSource:0}: Error finding container 24a1eec92d526113a346967bb4726720d1887f1793e47fbdafd4f6a41f5e2f52: Status 404 returned error can't find the container with id 24a1eec92d526113a346967bb4726720d1887f1793e47fbdafd4f6a41f5e2f52
Oct 30 20:06:39.930015 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:39.929978    5043 generic.go:356] "Generic (PLEG): container finished" podID="98db98c6-bca0-4bbc-9fdd-12912bbd1fd9" containerID="13547748f47654dd75975174395411fcf275c1d8f9fc7bd04490310de09ffe6a" exitCode=0
Oct 30 20:06:39.930015 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:39.930019    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-s5wgg" event={"ID":"98db98c6-bca0-4bbc-9fdd-12912bbd1fd9","Type":"ContainerDied","Data":"13547748f47654dd75975174395411fcf275c1d8f9fc7bd04490310de09ffe6a"}
Oct 30 20:06:39.930480 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:39.930071    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-s5wgg" event={"ID":"98db98c6-bca0-4bbc-9fdd-12912bbd1fd9","Type":"ContainerStarted","Data":"24a1eec92d526113a346967bb4726720d1887f1793e47fbdafd4f6a41f5e2f52"}
Oct 30 20:06:41.087153 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:41.087120    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-s5wgg"
Oct 30 20:06:41.097691 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:41.097659    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98db98c6-bca0-4bbc-9fdd-12912bbd1fd9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98db98c6-bca0-4bbc-9fdd-12912bbd1fd9-maas-api-xks-ca\") pod \"98db98c6-bca0-4bbc-9fdd-12912bbd1fd9\" (UID: \"98db98c6-bca0-4bbc-9fdd-12912bbd1fd9\") "
Oct 30 20:06:41.097877 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:41.097796    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98db98c6-bca0-4bbc-9fdd-12912bbd1fd9-kube-api-access-p6l7n\" (UniqueName: \"kubernetes.io/projected/98db98c6-bca0-4bbc-9fdd-12912bbd1fd9-kube-api-access-p6l7n\") pod \"98db98c6-bca0-4bbc-9fdd-12912bbd1fd9\" (UID: \"98db98c6-bca0-4bbc-9fdd-12912bbd1fd9\") "
Oct 30 20:06:41.097877 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:41.097852    5043 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98db98c6-bca0-4bbc-9fdd-12912bbd1fd9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98db98c6-bca0-4bbc-9fdd-12912bbd1fd9-maas-api-openshift-ca\") pod \"98db98c6-bca0-4bbc-9fdd-12912bbd1fd9\" (UID: \"98db98c6-bca0-4bbc-9fdd-12912bbd1fd9\") "
Oct 30 20:06:41.098215 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:41.098193    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98db98c6-bca0-4bbc-9fdd-12912bbd1fd9-maas-api-openshift-ca" pod "98db98c6-bca0-4bbc-9fdd-12912bbd1fd9" (UID: "98db98c6-bca0-4bbc-9fdd-12912bbd1fd9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:06:41.100105 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:41.100079    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98db98c6-bca0-4bbc-9fdd-12912bbd1fd9-maas-api-xks-ca" pod "98db98c6-bca0-4bbc-9fdd-12912bbd1fd9" (UID: "98db98c6-bca0-4bbc-9fdd-12912bbd1fd9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:06:41.100205 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:41.100190    5043 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98db98c6-bca0-4bbc-9fdd-12912bbd1fd9-kube-api-access-p6l7n" pod "98db98c6-bca0-4bbc-9fdd-12912bbd1fd9" (UID: "98db98c6-bca0-4bbc-9fdd-12912bbd1fd9"). InnerVolumeSpecName "kube-api-access-p6l7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:06:41.198963 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:41.198869    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98db98c6-bca0-4bbc-9fdd-12912bbd1fd9-maas-api-xks-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:06:41.198963 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:41.198905    5043 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6l7n\" (UniqueName: \"kubernetes.io/projected/98db98c6-bca0-4bbc-9fdd-12912bbd1fd9-kube-api-access-p6l7n\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:06:41.198963 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:41.198916    5043 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98db98c6-bca0-4bbc-9fdd-12912bbd1fd9-maas-api-openshift-ca\") on node \"ip-10-0-137-84.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 20:06:41.941891 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:41.941851    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-s5wgg" event={"ID":"98db98c6-bca0-4bbc-9fdd-12912bbd1fd9","Type":"ContainerDied","Data":"24a1eec92d526113a346967bb4726720d1887f1793e47fbdafd4f6a41f5e2f52"}
Oct 30 20:06:41.942115 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:41.941912    5043 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24a1eec92d526113a346967bb4726720d1887f1793e47fbdafd4f6a41f5e2f52"
Oct 30 20:06:41.942115 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:41.941870    5043 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-s5wgg"
Oct 30 20:06:49.662149 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:06:49.662107    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-b566bdfcc-ngn65_13cdaf87-6f93-41a7-aa9e-3adbb59839d6/authorino/0.log"
Oct 30 20:06:54.977810 ip-10-0-137-84 kubenswrapper[5043]: E1030 20:06:54.977767    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 20:07:03.041177 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:07:03.041139    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-6dsh9_e337bb42-030e-4061-828a-cb7a470e1564/revoke-keys/0.log"
Oct 30 20:07:03.773701 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:07:03.773667    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-s5wgg_98db98c6-bca0-4bbc-9fdd-12912bbd1fd9/revoke-keys/0.log"
Oct 30 20:07:03.947892 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:07:03.947857    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-fq9jj_cf450d63-f50f-490d-91b6-21c8cc1f0fb6/revoke-keys/0.log"
Oct 30 20:07:05.685663 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:07:05.685625    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-b566bdfcc-ngn65_13cdaf87-6f93-41a7-aa9e-3adbb59839d6/authorino/0.log"
Oct 30 20:07:05.914023 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:07:05.913990    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-wp2bs_4d6a8da0-155a-4d5c-b343-f9efea650d55/manager/0.log"
Oct 30 20:07:06.115878 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:07:06.115827    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-pbdbn_3d9b887e-6cc7-4f08-9365-38021d72e48f/manager/0.log"
Oct 30 20:07:06.519087 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:07:06.519047    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-4268w_1d0d721e-fbdc-4f18-88a2-094f7e42f9f8/manager/0.log"
Oct 30 20:07:06.932433 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:07:06.932349    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-9lkpq_8117eb3d-2697-4f54-bc43-e70935303423/manager/0.log"
Oct 30 20:07:07.736235 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:07:07.736202    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6455b4886fw65nx_0c9fc077-3f23-40d1-ba6e-2575551e01a8/istio-proxy/0.log"
Oct 30 20:07:07.934397 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:07:07.934363    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-zt46p_f15d5348-e5a7-44d7-9ba0-e1f58482ac21/discovery/0.log"
Oct 30 20:07:09.054535 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:07:09.054504    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-ftjzp_4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21/istio-proxy/1.log"
Oct 30 20:07:09.243548 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:07:09.243516    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-jrm57_cc14aeeb-67bf-438e-870e-595e528ba5c4/istio-proxy/1.log"
Oct 30 20:07:10.208138 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:07:10.208109    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-zgd28_8a958e54-98cc-4651-8c1b-dcb6b6207d1e/istio-proxy/0.log"
Oct 30 20:07:10.378306 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:07:10.378273    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h_0feb376c-3dc2-4831-a348-abaa09adade6/istio-proxy/0.log"
Oct 30 20:07:10.736032 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:07:10.735995    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-pqrwr_fa0d2e55-5431-43be-9c88-014880e5f1c0/payload-processing/0.log"
Oct 30 20:07:10.922302 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:07:10.922270    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-75866dc98c-4mnhf_247ff425-bed3-4297-b5c0-cc32b21bc179/router/0.log"
Oct 30 20:07:11.939335 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:07:11.939281    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-ftjzp_4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21/istio-proxy/0.log"
Oct 30 20:07:12.111507 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:07:12.111474    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-jrm57_cc14aeeb-67bf-438e-870e-595e528ba5c4/istio-proxy/0.log"
Oct 30 20:07:13.351211 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:07:13.351175    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-9426l_a57ae4e7-3e49-43ab-80a1-6db3d2c058d9/main/0.log"
Oct 30 20:07:13.721401 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:07:13.721372    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-zt5b2_a348d1da-3b22-4650-84a5-4e2de27d37e3/main/0.log"
Oct 30 20:07:14.080796 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:07:14.080686    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-6dg26_692c558e-cc23-4533-b6ad-0f982a82c187/main/0.log"
Oct 30 20:07:15.356117 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:07:15.356068    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-b566bdfcc-ngn65_13cdaf87-6f93-41a7-aa9e-3adbb59839d6/authorino/0.log"
Oct 30 20:07:24.976247 ip-10-0-137-84 kubenswrapper[5043]: E1030 20:07:24.976208    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 20:07:26.214009 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:07:26.213974    5043 scope.go:122] "RemoveContainer" containerID="55cf5dbafe8ed773eb8c62a4b9038ef40bb0b244210660c05f618586caaa87e5"
Oct 30 20:07:27.312537 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:07:27.312500    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-6dsh9_e337bb42-030e-4061-828a-cb7a470e1564/revoke-keys/0.log"
Oct 30 20:07:28.052361 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:07:28.052329    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-s5wgg_98db98c6-bca0-4bbc-9fdd-12912bbd1fd9/revoke-keys/0.log"
Oct 30 20:07:28.235406 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:07:28.235376    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-fq9jj_cf450d63-f50f-490d-91b6-21c8cc1f0fb6/revoke-keys/0.log"
Oct 30 20:07:29.601598 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:07:29.601557    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-b566bdfcc-ngn65_13cdaf87-6f93-41a7-aa9e-3adbb59839d6/authorino/0.log"
Oct 30 20:07:29.818162 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:07:29.818125    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-wp2bs_4d6a8da0-155a-4d5c-b343-f9efea650d55/manager/0.log"
Oct 30 20:07:30.011383 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:07:30.011352    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-pbdbn_3d9b887e-6cc7-4f08-9365-38021d72e48f/manager/0.log"
Oct 30 20:07:30.406761 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:07:30.406661    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-4268w_1d0d721e-fbdc-4f18-88a2-094f7e42f9f8/manager/0.log"
Oct 30 20:07:30.865085 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:07:30.865038    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-9lkpq_8117eb3d-2697-4f54-bc43-e70935303423/manager/0.log"
Oct 30 20:07:31.480788 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:07:31.480751    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6455b4886fw65nx_0c9fc077-3f23-40d1-ba6e-2575551e01a8/istio-proxy/0.log"
Oct 30 20:07:31.683214 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:07:31.683178    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-zt46p_f15d5348-e5a7-44d7-9ba0-e1f58482ac21/discovery/0.log"
Oct 30 20:07:32.840767 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:07:32.840716    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-ftjzp_4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21/istio-proxy/1.log"
Oct 30 20:07:33.040113 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:07:33.040079    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-jrm57_cc14aeeb-67bf-438e-870e-595e528ba5c4/istio-proxy/1.log"
Oct 30 20:07:34.072433 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:07:34.072395    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-zgd28_8a958e54-98cc-4651-8c1b-dcb6b6207d1e/istio-proxy/0.log"
Oct 30 20:07:34.283287 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:07:34.283256    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h_0feb376c-3dc2-4831-a348-abaa09adade6/istio-proxy/0.log"
Oct 30 20:07:34.652001 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:07:34.651971    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-pqrwr_fa0d2e55-5431-43be-9c88-014880e5f1c0/payload-processing/0.log"
Oct 30 20:07:34.834173 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:07:34.834146    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-75866dc98c-4mnhf_247ff425-bed3-4297-b5c0-cc32b21bc179/router/0.log"
Oct 30 20:07:35.598972 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:07:35.598945    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-9426l_a57ae4e7-3e49-43ab-80a1-6db3d2c058d9/main/0.log"
Oct 30 20:07:35.999861 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:07:35.999830    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-zt5b2_a348d1da-3b22-4650-84a5-4e2de27d37e3/main/0.log"
Oct 30 20:07:36.369988 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:07:36.369899    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-6dg26_692c558e-cc23-4533-b6ad-0f982a82c187/main/0.log"
Oct 30 20:07:54.975106 ip-10-0-137-84 kubenswrapper[5043]: E1030 20:07:54.975068    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 20:08:16.593326 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:08:16.593239    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-pqrwr_fa0d2e55-5431-43be-9c88-014880e5f1c0/payload-processing/0.log"
Oct 30 20:08:16.771137 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:08:16.771070    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-pqrwr_fa0d2e55-5431-43be-9c88-014880e5f1c0/payload-processing/0.log"
Oct 30 20:08:24.975826 ip-10-0-137-84 kubenswrapper[5043]: E1030 20:08:24.975786    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 20:08:26.235680 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:08:26.235641    5043 scope.go:122] "RemoveContainer" containerID="da24ea850abc368fbeddce9b985922f40ef5907f93989ebe850f4376bb1a0276"
Oct 30 20:08:54.975607 ip-10-0-137-84 kubenswrapper[5043]: E1030 20:08:54.975570    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 20:09:03.926528 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:03.926490    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-pqrwr_fa0d2e55-5431-43be-9c88-014880e5f1c0/payload-processing/0.log"
Oct 30 20:09:04.105180 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:04.105147    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-pqrwr_fa0d2e55-5431-43be-9c88-014880e5f1c0/payload-processing/0.log"
Oct 30 20:09:07.897207 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:07.897171    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wg2df_6ecb643b-1dce-4523-89df-93b22ad8ce6f/global-pull-secret-syncer/0.log"
Oct 30 20:09:07.939487 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:07.939456    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-65kc2_9201f6f2-73b9-45cd-9950-3a07914d8ea9/connection-checker/0.log"
Oct 30 20:09:07.966665 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:07.966637    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-6b8sd_00b97df0-857e-4ad9-b33a-4a59777b6f62/connection-checker/0.log"
Oct 30 20:09:07.993478 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:07.993450    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-zqvrw_c84a5c22-7a46-4de1-aef7-32cb9a5453bb/connection-checker/0.log"
Oct 30 20:09:08.141839 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:08.141811    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lqmqr_2d261662-727d-4d68-9f4a-9790ecf16d46/konnectivity-agent/0.log"
Oct 30 20:09:08.179107 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:08.179022    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-84.us-east-2.compute.internal_532f06e8e868303658c6aeb582644516/haproxy/0.log"
Oct 30 20:09:14.923748 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:14.923707    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-b566bdfcc-ngn65_13cdaf87-6f93-41a7-aa9e-3adbb59839d6/authorino/0.log"
Oct 30 20:09:14.982731 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:14.982682    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-wp2bs_4d6a8da0-155a-4d5c-b343-f9efea650d55/manager/0.log"
Oct 30 20:09:15.021164 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:15.021131    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-pbdbn_3d9b887e-6cc7-4f08-9365-38021d72e48f/manager/0.log"
Oct 30 20:09:15.146222 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:15.146189    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-4268w_1d0d721e-fbdc-4f18-88a2-094f7e42f9f8/manager/0.log"
Oct 30 20:09:15.328525 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:15.328493    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-9lkpq_8117eb3d-2697-4f54-bc43-e70935303423/manager/0.log"
Oct 30 20:09:16.892821 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:16.892788    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-2sdkw_06397fd4-3377-4eaa-8879-ebbb5943fbda/cluster-monitoring-operator/0.log"
Oct 30 20:09:17.283090 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:17.283059    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nrt89_0359a57a-4c15-4eba-bc01-7d5b5d3d97bf/node-exporter/0.log"
Oct 30 20:09:17.330492 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:17.330460    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nrt89_0359a57a-4c15-4eba-bc01-7d5b5d3d97bf/kube-rbac-proxy/0.log"
Oct 30 20:09:17.375877 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:17.375855    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nrt89_0359a57a-4c15-4eba-bc01-7d5b5d3d97bf/init-textfile/0.log"
Oct 30 20:09:17.631703 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:17.631626    5043 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-x9glm/perf-node-gather-daemonset-ptfnx"]
Oct 30 20:09:17.632120 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:17.632099    5043 cpu_manager.go:414] "containerMap: removing container" podUID="98db98c6-bca0-4bbc-9fdd-12912bbd1fd9" containerName="revoke-keys"
Oct 30 20:09:17.632295 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:17.632218    5043 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98db98c6-bca0-4bbc-9fdd-12912bbd1fd9" containerName="revoke-keys"
Oct 30 20:09:17.632437 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:17.632420    5043 memory_manager.go:365] "RemoveStaleState removing state" podUID="98db98c6-bca0-4bbc-9fdd-12912bbd1fd9" containerName="revoke-keys"
Oct 30 20:09:17.637764 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:17.637711    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-ptfnx"
Oct 30 20:09:17.647982 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:17.647952    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x9glm/perf-node-gather-daemonset-ptfnx"]
Oct 30 20:09:17.649849 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:17.649823    5043 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-x9glm\"/\"default-dockercfg-fzhm4\""
Oct 30 20:09:17.650059 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:17.649873    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x9glm\"/\"kube-root-ca.crt\""
Oct 30 20:09:17.650622 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:17.650603    5043 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x9glm\"/\"openshift-service-ca.crt\""
Oct 30 20:09:17.741970 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:17.741932    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md89t\" (UniqueName: \"kubernetes.io/projected/af00767c-688e-408e-baba-516a97ec39a5-kube-api-access-md89t\") pod \"perf-node-gather-daemonset-ptfnx\" (UID: \"af00767c-688e-408e-baba-516a97ec39a5\") " pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-ptfnx"
Oct 30 20:09:17.742174 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:17.742071    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/af00767c-688e-408e-baba-516a97ec39a5-podres\") pod \"perf-node-gather-daemonset-ptfnx\" (UID: \"af00767c-688e-408e-baba-516a97ec39a5\") " pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-ptfnx"
Oct 30 20:09:17.742174 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:17.742165    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af00767c-688e-408e-baba-516a97ec39a5-sys\") pod \"perf-node-gather-daemonset-ptfnx\" (UID: \"af00767c-688e-408e-baba-516a97ec39a5\") " pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-ptfnx"
Oct 30 20:09:17.742355 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:17.742331    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/af00767c-688e-408e-baba-516a97ec39a5-proc\") pod \"perf-node-gather-daemonset-ptfnx\" (UID: \"af00767c-688e-408e-baba-516a97ec39a5\") " pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-ptfnx"
Oct 30 20:09:17.742400 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:17.742368    5043 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/af00767c-688e-408e-baba-516a97ec39a5-lib-modules\") pod \"perf-node-gather-daemonset-ptfnx\" (UID: \"af00767c-688e-408e-baba-516a97ec39a5\") " pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-ptfnx"
Oct 30 20:09:17.843170 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:17.843131    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/af00767c-688e-408e-baba-516a97ec39a5-podres\") pod \"perf-node-gather-daemonset-ptfnx\" (UID: \"af00767c-688e-408e-baba-516a97ec39a5\") " pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-ptfnx"
Oct 30 20:09:17.843365 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:17.843196    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af00767c-688e-408e-baba-516a97ec39a5-sys\") pod \"perf-node-gather-daemonset-ptfnx\" (UID: \"af00767c-688e-408e-baba-516a97ec39a5\") " pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-ptfnx"
Oct 30 20:09:17.843365 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:17.843233    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/af00767c-688e-408e-baba-516a97ec39a5-proc\") pod \"perf-node-gather-daemonset-ptfnx\" (UID: \"af00767c-688e-408e-baba-516a97ec39a5\") " pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-ptfnx"
Oct 30 20:09:17.843365 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:17.843256    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/af00767c-688e-408e-baba-516a97ec39a5-lib-modules\") pod \"perf-node-gather-daemonset-ptfnx\" (UID: \"af00767c-688e-408e-baba-516a97ec39a5\") " pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-ptfnx"
Oct 30 20:09:17.843365 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:17.843286    5043 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md89t\" (UniqueName: \"kubernetes.io/projected/af00767c-688e-408e-baba-516a97ec39a5-kube-api-access-md89t\") pod \"perf-node-gather-daemonset-ptfnx\" (UID: \"af00767c-688e-408e-baba-516a97ec39a5\") " pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-ptfnx"
Oct 30 20:09:17.843365 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:17.843290    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/af00767c-688e-408e-baba-516a97ec39a5-podres\") pod \"perf-node-gather-daemonset-ptfnx\" (UID: \"af00767c-688e-408e-baba-516a97ec39a5\") " pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-ptfnx"
Oct 30 20:09:17.843365 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:17.843320    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af00767c-688e-408e-baba-516a97ec39a5-sys\") pod \"perf-node-gather-daemonset-ptfnx\" (UID: \"af00767c-688e-408e-baba-516a97ec39a5\") " pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-ptfnx"
Oct 30 20:09:17.843580 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:17.843361    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/af00767c-688e-408e-baba-516a97ec39a5-proc\") pod \"perf-node-gather-daemonset-ptfnx\" (UID: \"af00767c-688e-408e-baba-516a97ec39a5\") " pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-ptfnx"
Oct 30 20:09:17.843580 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:17.843391    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/af00767c-688e-408e-baba-516a97ec39a5-lib-modules\") pod \"perf-node-gather-daemonset-ptfnx\" (UID: \"af00767c-688e-408e-baba-516a97ec39a5\") " pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-ptfnx"
Oct 30 20:09:17.853077 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:17.853044    5043 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md89t\" (UniqueName: \"kubernetes.io/projected/af00767c-688e-408e-baba-516a97ec39a5-kube-api-access-md89t\") pod \"perf-node-gather-daemonset-ptfnx\" (UID: \"af00767c-688e-408e-baba-516a97ec39a5\") " pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-ptfnx"
Oct 30 20:09:17.954551 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:17.954463    5043 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-ptfnx"
Oct 30 20:09:18.035923 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:18.035897    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-bbdvn_05d9b81e-3c5a-4590-ba33-6598855a9a5e/prometheus-operator-admission-webhook/0.log"
Oct 30 20:09:18.234760 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:18.234710    5043 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x9glm/perf-node-gather-daemonset-ptfnx"]
Oct 30 20:09:18.236726 ip-10-0-137-84 kubenswrapper[5043]: W1030 20:09:18.236700    5043 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podaf00767c_688e_408e_baba_516a97ec39a5.slice/crio-1141dc47288c0afe4ad989212aaa943b4b9349429d0ae85f2df42fd572a2046d WatchSource:0}: Error finding container 1141dc47288c0afe4ad989212aaa943b4b9349429d0ae85f2df42fd572a2046d: Status 404 returned error can't find the container with id 1141dc47288c0afe4ad989212aaa943b4b9349429d0ae85f2df42fd572a2046d
Oct 30 20:09:18.842623 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:18.842580    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-ptfnx" event={"ID":"af00767c-688e-408e-baba-516a97ec39a5","Type":"ContainerStarted","Data":"adaf3403291c80c997ddf9a8e4a8125137803c926dc215527b8e75faec030b36"}
Oct 30 20:09:18.842862 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:18.842650    5043 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-ptfnx" event={"ID":"af00767c-688e-408e-baba-516a97ec39a5","Type":"ContainerStarted","Data":"1141dc47288c0afe4ad989212aaa943b4b9349429d0ae85f2df42fd572a2046d"}
Oct 30 20:09:18.842862 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:18.842678    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-ptfnx"
Oct 30 20:09:18.861191 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:18.861136    5043 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-ptfnx" podStartSLOduration=1.861120944 podStartE2EDuration="1.861120944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:09:18.860150152 +0000 UTC m=+3178.371210129" watchObservedRunningTime="2026-10-30 20:09:18.861120944 +0000 UTC m=+3178.372180943"
Oct 30 20:09:19.688236 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:19.688198    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-lpg7w_a85cca66-a438-4117-9f50-977c7f528b7c/networking-console-plugin/0.log"
Oct 30 20:09:20.315098 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:20.315068    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cmw6w_76e21916-0744-4499-a0cb-64a7a44a8285/console-operator/2.log"
Oct 30 20:09:20.323486 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:20.323455    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cmw6w_76e21916-0744-4499-a0cb-64a7a44a8285/console-operator/3.log"
Oct 30 20:09:20.861465 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:20.861417    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-wh4j2_cb415120-7b29-4553-ba70-cf2b3e00305e/download-server/0.log"
Oct 30 20:09:21.487915 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:21.487882    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-mg98j_c243f1d8-1c43-4eb8-93ec-cb28d72cc8a1/volume-data-source-validator/0.log"
Oct 30 20:09:22.849001 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:22.848957    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gkn6q_0269ffa6-ce06-43f3-be2a-58f6aa3e6964/dns/0.log"
Oct 30 20:09:22.870935 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:22.870905    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gkn6q_0269ffa6-ce06-43f3-be2a-58f6aa3e6964/kube-rbac-proxy/0.log"
Oct 30 20:09:22.971170 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:22.971144    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xjjqh_36ab52a2-65e2-4af6-a872-9052e12714d1/dns-node-resolver/0.log"
Oct 30 20:09:23.620305 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:23.620268    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-f45b6f4dd-7rkpk_02190f18-2bb0-4f10-8a37-776da4e54aa3/registry/0.log"
Oct 30 20:09:23.716191 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:23.716156    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zj8d6_f1ea27b2-80df-404c-ac39-460f495957b2/node-ca/0.log"
Oct 30 20:09:24.775448 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:24.775419    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6455b4886fw65nx_0c9fc077-3f23-40d1-ba6e-2575551e01a8/istio-proxy/0.log"
Oct 30 20:09:24.861911 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:24.861885    5043 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-ptfnx"
Oct 30 20:09:24.985728 ip-10-0-137-84 kubenswrapper[5043]: E1030 20:09:24.985684    5043 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-smgf6" containerName="extractor"
Oct 30 20:09:27.111782 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:27.111745    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-zt46p_f15d5348-e5a7-44d7-9ba0-e1f58482ac21/discovery/0.log"
Oct 30 20:09:27.314427 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:27.314382    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-ftjzp_4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21/istio-proxy/1.log"
Oct 30 20:09:27.314722 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:27.314693    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-ftjzp_4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21/istio-proxy/0.log"
Oct 30 20:09:27.343704 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:27.343676    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-jrm57_cc14aeeb-67bf-438e-870e-595e528ba5c4/istio-proxy/1.log"
Oct 30 20:09:27.343925 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:27.343910    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-jrm57_cc14aeeb-67bf-438e-870e-595e528ba5c4/istio-proxy/0.log"
Oct 30 20:09:27.506213 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:27.506184    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-zgd28_8a958e54-98cc-4651-8c1b-dcb6b6207d1e/istio-proxy/0.log"
Oct 30 20:09:27.534345 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:27.534310    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h_0feb376c-3dc2-4831-a348-abaa09adade6/istio-proxy/0.log"
Oct 30 20:09:27.545587 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:27.545545    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6455b4886fw65nx_0c9fc077-3f23-40d1-ba6e-2575551e01a8/istio-proxy/0.log"
Oct 30 20:09:27.591500 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:27.591412    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-pqrwr_fa0d2e55-5431-43be-9c88-014880e5f1c0/payload-processing/0.log"
Oct 30 20:09:27.628492 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:27.628456    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-75866dc98c-4mnhf_247ff425-bed3-4297-b5c0-cc32b21bc179/router/0.log"
Oct 30 20:09:29.598302 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:29.598248    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9jkcb_765d8d9d-b89f-419f-b237-3f8034db1bed/serve-healthcheck-canary/0.log"
Oct 30 20:09:29.905238 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:29.905152    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-zt46p_f15d5348-e5a7-44d7-9ba0-e1f58482ac21/discovery/0.log"
Oct 30 20:09:30.007469 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:30.007383    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-ftjzp_4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21/istio-proxy/1.log"
Oct 30 20:09:30.007651 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:30.007527    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-ftjzp_4d17d9d4-53d0-4f8e-87a1-3b43f94d2b21/istio-proxy/0.log"
Oct 30 20:09:30.025861 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:30.025825    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-jrm57_cc14aeeb-67bf-438e-870e-595e528ba5c4/istio-proxy/1.log"
Oct 30 20:09:30.026156 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:30.026130    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-jrm57_cc14aeeb-67bf-438e-870e-595e528ba5c4/istio-proxy/0.log"
Oct 30 20:09:30.111288 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:30.111211    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-zgd28_8a958e54-98cc-4651-8c1b-dcb6b6207d1e/istio-proxy/0.log"
Oct 30 20:09:30.115391 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:30.115367    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-fz9hf_e5002691-8257-4dab-a88b-3c1b60e8e6f8/insights-operator/0.log"
Oct 30 20:09:30.116775 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:30.116753    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-fz9hf_e5002691-8257-4dab-a88b-3c1b60e8e6f8/insights-operator/1.log"
Oct 30 20:09:30.127660 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:30.127638    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-64695b4bc9-qmb5h_0feb376c-3dc2-4831-a348-abaa09adade6/istio-proxy/0.log"
Oct 30 20:09:30.159338 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:30.159262    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-pqrwr_fa0d2e55-5431-43be-9c88-014880e5f1c0/payload-processing/0.log"
Oct 30 20:09:30.182481 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:30.182451    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-75866dc98c-4mnhf_247ff425-bed3-4297-b5c0-cc32b21bc179/router/0.log"
Oct 30 20:09:30.389958 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:30.389928    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-smgf6_aeef13dc-e06e-49c5-b893-64c7f73a7e44/kube-rbac-proxy/0.log"
Oct 30 20:09:30.443193 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:30.443118    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-smgf6_aeef13dc-e06e-49c5-b893-64c7f73a7e44/exporter/0.log"
Oct 30 20:09:30.472694 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:30.472664    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-smgf6_aeef13dc-e06e-49c5-b893-64c7f73a7e44/extractor/0.log"
Oct 30 20:09:30.520838 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:30.520811    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-tqp8q-5cb4z_030d88d7-44c3-4945-923e-1e57c2ffd18e/insights-gathering/0.log"
Oct 30 20:09:33.028024 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:33.027991    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6455b4886fw65nx_0c9fc077-3f23-40d1-ba6e-2575551e01a8/istio-proxy/0.log"
Oct 30 20:09:33.157013 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:33.156983    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-68fb5c5984-svlrj_3d93bc51-b50b-4595-b6c5-7494bfaec7a9/manager/0.log"
Oct 30 20:09:33.193914 ip-10-0-137-84 kubenswrapper[5043]: I1030 20:09:33.193880    5043 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-68fb5c5984-svlrj_3d93bc51-b50b-4595-b6c5-7494bfaec7a9/copy-manifests/0.log"